Qwen3.5-9B — GGUF (imatrix) weight-quant set

A complete set of importance-matrix (imatrix) GGUF quantizations of Qwen/Qwen3.5-9B, spanning the full ladder from IQ1_* up to Q8_0, plus the BF16 baseline. This set exists as a canonical reference for cross-quant / cross-backend quality (perplexity) and throughput comparison.

Every quant in this repository was produced from the same BF16 source and the same importance matrix (Qwen3.5-9B-imatrix.gguf, included here for full reproducibility).

Generated with the jimbothigpen/llama.cpp fork. These GGUFs and their imatrix are produced by that fork's quantization toolchain. It is a fork of upstream ggml-org/llama.cpp that integrates features ported from several llama.cpp forks — including the IQ-K / IQ-KS / IQ-KT (trellis) quant families from ik_llama.cpp — plus ROCm and Vulkan backend support.

Provenance

These are derivative weights of Qwen/Qwen3.5-9B and inherit its Apache-2.0 license (LICENSE).

Importance matrix (imatrix)

The imatrix steers per-weight quantization error toward the activations that matter, materially improving low-bit quality. It was computed once over the full calibration corpus and reused for all quants.

• Calibration corpus: calibration_datav3 — the de-facto community-standard llama.cpp imatrix calibration set (lineage: kalomaze groups_merged → Dampf → bartowski calibration_datav3). It is a semantic, multi-domain text mix deliberately disjoint from any perplexity evaluation set (e.g. wikitext-2), so imatrix calibration does not contaminate downstream PPL measurement.
• Coverage: the entire corpus (no chunk cap).
• Command (settings):

  llama-imatrix -m Qwen3.5-9B-BF16.gguf -f calibration_datav3.txt \
                -o Qwen3.5-9B-imatrix.gguf \
                -ngl 99 -fa on -fit off --no-mmap -b 512
  

The Qwen3.5-9B-imatrix.gguf file is included in this repo — you can reuse it to re-quantize from the BF16 source and reproduce any file here bit-for-bit (given the same quantizer build).

Quantization settings

Each quant was produced directly from the BF16 GGUF with the shared imatrix applied:

llama-quantize --imatrix Qwen3.5-9B-imatrix.gguf \
               Qwen3.5-9B-BF16.gguf Qwen3.5-9B-<TYPE>.gguf <TYPE>

• imatrix is applied to all types in this set except WHT3_0 and WHT4_0 (see WHT quants below for why those two are intentionally unweighted).
• No per-tensor type overrides beyond each quant type's own built-in policy.
• Ternary-only types (TQ1_0/TQ2_0) are intentionally excluded — they are not meaningful for a non-ternary base model.

Files

BF16 baseline + 41 quant types + mmproj:

WHT (Walsh-Hadamard-rotated, no imatrix): WHT3_0 (4.0 bpw) · WHT4_0 (5.0 bpw)

IQ-K / trellis family (quant formats ported from ik_llama.cpp — needs a compatible build, see Compatibility): IQ1_KT · IQ2_K · IQ2_KS · IQ2_KL · IQ2_KT · IQ3_K · IQ3_KS · IQ3_KT · IQ4_K · IQ4_KS · IQ4_KSS · IQ4_KT · IQ5_K · IQ5_KS · IQ6_K

IQ (mainline): IQ1_S · IQ1_M · IQ2_XXS · IQ2_XS · IQ2_S · IQ2_M · IQ3_XS · IQ3_M · IQ4_XS · IQ4_NL

K-quants: Q2_K · Q3_K_S · Q3_K_M · Q3_K_L · Q4_K_S · Q4_K_M · Q5_K_S · Q5_K_M · Q6_K

Legacy: Q4_0 · Q4_1 · Q5_0 · Q5_1 · Q8_0

Multimodal: Qwen3.5-9B-mmproj-F16.gguf (vision projector — use with any language-model quant above)

Compatibility

• The mainline types (Q2_K…Q8_0, IQ1_S…IQ4_NL) load in standard llama.cpp.
• The IQ-K / “KS” / “KT” (trellis) / “KL” / “KSS” types are ik_llama.cpp-family quants and require an ik_llama.cpp-compatible build to load and run.
• The WHT types (WHT3_0, WHT4_0) require a build of the jimbothigpen/llama.cpp fork (or another build that includes the WHT quant family).

WHT (Walsh-Hadamard-rotated) quants — WHT3_0 and WHT4_0

A fast Walsh-Hadamard / random-Hadamard transform is applied per 32-weight block before quantizing. This rotation flattens per-column outliers, redistributing the weight distribution toward the N(0,1) assumption the codebook was designed for, which improves low-bit fidelity.

These are built WITHOUT an importance matrix — and that is deliberate. The block rotation mixes the original input columns, so an importance vector computed in the original basis is no longer aligned with the rotated coefficients. Applying imatrix weighting to the rotated residual measurably hurts PPL:

Qwen3.5-9B, wikitext PPL: WHT3_0 unweighted = 7.27 vs imatrix-weighted = 8.61 (+15.5%). The unweighted WHT3_0 also edges out the unweighted TheTom reference (~7.68).

Do NOT supply an imatrix when producing or re-quantizing WHT types — the quantizer ignores it by design, but passing one is wasted effort and the round-trip will not reproduce the files here.

Attribution. The WHT (Walsh–Hadamard–rotated) quantization method originates with TheTom (llama-cpp-turboquant). WHT3_0 / WHT4_0 use the same block byte-layout as TheTom's TQ3_1S / TQ4_1S rotated-quant types, but are assigned distinct ggml_type IDs (80/81), so these GGUFs load only with this fork — not with TheTom's llama-cpp-turboquant build. They are quantized unweighted per TheTom's reference implementation — credit for the underlying rotated-quant algorithm goes to TheTom.

Multimodal — Qwen3.5-9B-mmproj-F16.gguf (vision projector)

Qwen3.5-9B is a multimodal (vision) model. This repo includes the high-precision vision projector Qwen3.5-9B-mmproj-F16.gguf, which enables image input at runtime.

Pair it with any language-model quant from this repo:

# CLI
llama-mtmd-cli \
    --mmproj Qwen3.5-9B-mmproj-F16.gguf \
    -m Qwen3.5-9B-<TYPE>.gguf \
    -p "Describe this image." --image /path/to/image.jpg

# Server
llama-server \
    --mmproj Qwen3.5-9B-mmproj-F16.gguf \
    -m Qwen3.5-9B-<TYPE>.gguf

The mmproj file is the same for every quant of this model — download it once and combine it with whichever language-model GGUF you choose (e.g. Q4_K_M, IQ4_KT, WHT4_0, etc.).

Reproducing / verification

The imatrix and BF16 source are both present, so any quant here can be regenerated and verified against its file. Perplexity for these quants is measured against wikitext-2-raw (a set disjoint from the calibration corpus above).