Qwen3.5-9B-Uncensored-Genesis — NVFP4 imatrix GGUF

NVFP4 quantisations of LuffyTheFox/Qwen3.5-9B-Uncensored-Genesis-BF16-GGUF, built with michaelw9999/advanced-gguf-quantizer (a llama.cpp fork focused on NVFP4/MXFP6 quantization).

Files

v3 — Recommended (correct pipeline)

v2 / original — Deprecated (flawed pipeline)

The v2/original files used a --tensor-type .*=nvfp4 catch-all that wrongly forced output.weight to NVFP4 (should be Q6_K) and all FFN tensors to NVFP4 (should stay F16). The v3 files use the correct pipeline — see the Pipeline fix section below.

Source

Requantised from BF16 — no extra rounding step from a lower-precision source.

Pipeline fix (v2 → v3)

The v2/original files were quantized with a --tensor-type .*=nvfp4 catch-all that forced every eligible tensor to NVFP4, overriding the default NVFP4 policy that keeps sensitive tensors at higher precision.

The v3 files drop the catch-all and use NVFP4 as the output type, letting the default policy protect sensitive tensors:

The v3 files are slightly larger (6.53 GiB vs 5.3 GB) because Q6_K and F16 tensors take more space than NVFP4, but the output projection and embedding quality is preserved.

Imatrix variants

NVFP4 uses per-block input scales. Without an imatrix those scales fall back to identity (all 1.0). An imatrix pass measures how much each weight influences activations on real text, and the quantiser spends the input scale budget where it matters.

v5 — plain text. Bartowski's v5 calibration set, ~1.7 MB of plain prose. Broad language coverage, no chat-template structure. 256 chunks.

v6 — chat-template rendered. Bartowski prose plus 173 conversations (137 base + 36 tool-calling) rendered through the Qwen3.5-9B native chat template (Qwen/Qwen3.5-9B). Rendering preserves the special/control tokens and role structure the model sees at inference time, so the activation statistics better match chat/tool-use workloads. llama-imatrix was run with --parse-special so the rendered special-token markup is treated as real tokens, not literal text. 256 chunks.

For chat, reasoning, or tool calling: use v6. For plain completion or broad text work: v5 is fine. v3 (no imatrix) is the baseline.

No PPL/KLD measurements are provided. Quantized-model perplexity and KLD against the BF16 source have not been measured. The imatrix PPL values reported by llama-imatrix during calibration reflect the source model's perplexity on the calibration text, not the quantized model's quality. They are not comparable across variants and are omitted for that reason.

Tensor protection policy

The protection policy follows Luffy's per-tensor quant policy, mapped from the MoE Qwen 3.6 35B (ffn_down_exps) to this dense model (ffn_down).

F16 singular-collapse protection:

F32 architecture-specific protection:

Forced NVFP4 (do not push lower, collapses):

Default NVFP4 policy (no catch-all):

The v3 pipeline uses NVFP4 as the output type without a .*=nvfp4 catch-all. The default NVFP4 policy keeps these at higher precision:

• output.weight → Q6_K
• token_embd.weight → F16

Everything else eligible takes NVFP4.

Tensor mix (v3)

• general.file_type = 39 (NVFP4)
• qwen35.block_count = 32
• No nextn_predict_layers (no MTP)
• No unexpected 1D norm .scale tensors

Usage

llama-cli -m Qwen3.5-9B-Uncensored-Genesis-NVFP4-v3-imx-v6.gguf \
  -ngl 99 -c 4096 --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0

Sampling follows Genesis guidance (temp 0.6, top_k 20, top_p and min_p disabled). For code, also set --repeat-penalty 1.0 — code repeats identifiers by nature, and any penalty pushes the model off the identifier it just chose.

Vision

Qwen3.5-9B is a vision-language model. The text model is quantised here; the multimodal projection (mmproj-Qwen3.5-9B-Uncensored-Genesis-BF16.gguf) is available in the source repo and should be used unmodified alongside these weights.

Hardware

• Blackwell (RTX 50xx): native FP4 path, fastest. Build llama.cpp with BLACKWELL_NATIVE_FP4 = 1.
• Ampere (RTX 30xx): NVFP4 inference works via fallback kernels. The F32 SSM protections prevent a CUDA assert in the patched SSM convolution kernel on this architecture.
• Quantisation was done CPU-only because the Ampere CUDA NVFP4 encoder is inefficient (hangs/spins at 99% GPU usage with ~300MB VRAM).

Reproducibility

v5 imatrix

llama-imatrix \
  -m Qwen3.5-9B-Uncensored-Genesis-BF16.gguf \
  -f calibration_datav5.txt \
  -o imatrix_v5.dat \
  --output-format dat \
  -ngl all -b 2048 -ub 512 -t 6 --chunks 256

v6 imatrix

Conversations rendered through the Qwen3.5-9B native chat template with transformers.AutoTokenizer.apply_chat_template(..., tools=tools, add_generation_prompt=False, tokenize=False), concatenated with prose, fed to llama-imatrix with --parse-special:

llama-imatrix \
  -m Qwen3.5-9B-Uncensored-Genesis-BF16.gguf \
  -f calibration_v6_rendered.txt \
  -o imatrix_v6.dat \
  --output-format dat \
  -ngl all -b 2048 -ub 512 -t 6 --chunks 256 \
  --parse-special

Quantisation (v3 — correct pipeline)

All v3 variants use the same protection policy and CPU-only quantise command, differing only in the imatrix file (or its absence for v3 baseline):

llama-quantize \
  --allow-requantize \
  --tensor-type-file tensor_types_protection.txt \
  --imatrix imatrix_v6.dat \
  Qwen3.5-9B-Uncensored-Genesis-BF16.gguf \
  Qwen3.5-9B-Uncensored-Genesis-NVFP4-v3-imx-v6.gguf \
  NVFP4 6

Key differences from the v2 pipeline:

• Uses NVFP4 as the output type (not Q8_0 with .*=nvfp4 catch-all)
• Uses --tensor-type-file for per-tensor overrides (not inline --tensor-type regex)
• No --mode fast (unnecessary, only affects selector)
• The default NVFP4 policy keeps output.weight at Q6_K and token_embd.weight at F16

Credits

• Base model and finetune: LuffyTheFox
• Per-tensor protection policy: LuffyTheFox
• Chat template for v6 calibration: Qwen/Qwen3.5-9B native template
• Quantiser: michaelw9999/advanced-gguf-quantizer
• Calibration corpus: Bartowski v5 plain text; v6 prose + conversations rendered through the Qwen3.5-9B chat template