Qwen3-Reranker-0.6B — GGUF (NVFP4 / Q6_K)
GGUF quantizations of Qwen/Qwen3-Reranker-0.6B for
llama.cpp, built for the
llama-server --reranking / /rerank path. These are hybrid quantizations: the 2-way
relevance head (cls.output.weight) is kept in BF16, every normalization tensor is kept in
F32, and in the NVFP4 build the 155M-row embedding table is kept in Q8_0 — so the
yes/no scoring path and the embedding lookup stay at high precision while the bulk of the 0.6B
body is compressed. The result is files 2.4–2.9× smaller than the BF16 reference with
essentially unchanged ranking scores.
Files
| File | Size | Body quant | Embed / head / norms | eff. bpw | vs BF16 |
|---|---|---|---|---|---|
Qwen3-Reranker-0.6B.NVFP4.gguf | 399.5 MiB | NVFP4 | Q8_0 embed, BF16 head, F32 norms | 5.63 | 2.86× smaller (−742.6 MiB) |
Qwen3-Reranker-0.6B.Q6_K.gguf | 472.0 MiB | Q6_K | Q6_K embed, BF16 head, F32 norms | 6.65 | 2.42× smaller (−670.2 MiB) |
91bbd34601c7b8ddb8a8413cdea90536e9754c010da0be7112246d7441411c7d Qwen3-Reranker-0.6B.NVFP4.gguf
4e0cb92dd4e1f1c8eb6d9537f95d14b7f89aa48da27d64dc826e5837be2c6b40 Qwen3-Reranker-0.6B.Q6_K.gguf
Quantization layout
Both files are GGUF v3, architecture qwen3, 311 tensors, 595,778,560 parameters
(595.8M). The NVFP4 file is a deliberate per‑tensor mapping, not a vanilla
llama-quantize X NVFP4 output:
| Tensors | Count | NVFP4 | Q6_K | Notes |
|---|---|---|---|---|
| Attention/FFN matmuls (28 blocks × 7) | 196 | NVFP4 | Q6_K | the compressed body (≈ 4.47 / ≈ 6.53 bpw) |
token_embd.weight | 1 | Q8_0 | Q6_K | embed table, 1024 × 151,669 — kept at 8.50 bpw in the NVFP4 build |
cls.output.weight | 1 | BF16 | BF16 | 1024 × 2 — the relevance head that emits the yes/no score |
All norms (attn_norm, ffn_norm, attn_q_norm, attn_k_norm × 28 + output_norm) | 113 | F32 | F32 | every normalization tensor left at full precision |
The design keeps the tensors that most directly shape the final score — the embedding lookup, the 2‑way head, and every norm — at high precision and compresses only the large linear layers. In the NVFP4 build that costs 157.4 MiB for the Q8_0 embed table (vs an all‑NVFP4 build at 325.5 MiB) in exchange for keeping the lookup path out of the NVFP4 mid‑band distortion regime.
Fidelity vs BF16
Q6_K — controlled harness
Measured against a BF16 GGUF reference over 12 queries × 5 passages = 60 reranks
(2 relevant, 1 near‑miss, 2 off‑topic each), CPU, 32 threads, flash‑attention on, ctx 1024,
no prompt caching. Prompts are built from the identical embedded rerank template, so the
comparison isolates quantization error.
| Metric (vs BF16) | Q6 |
|---|---|
| Spearman ρ (all 60) | 0.9962 |
| Kendall τ (all 60) | 0.9616 |
| Kendall τ (avg per query) | 0.9500 |
| mean |Δscore| | 0.0022 |
| max |Δscore| | 0.091 |
| Full top‑5 ordering identical | 9/12 |
| Per‑query Spearman ρ (avg / min) | 0.9750 / 0.900 |
| Mean absolute rank shift per doc | 0.100 |
| Decision match at 0.5 threshold | 60/60 |
| Top‑1 doc match | 11/12 |
Q6 is effectively indistinguishable from BF16 on this workload — deviations are at most a single adjacent swap, and never across the relevant/off‑topic boundary.
NVFP4 (V1) — live memory bank
Measured against the same BF16 reference on a live Hindsight memory bank: 18 queries ×
269 stored documents, served through the production llama-server --reranking path.
| Comparison | Kendall τ | median |Δrank| | what it isolates |
|---|---|---|---|
| Q8_0 (CPU) vs BF16 (CPU) | 0.97 | 3.0 | quantization only, same device |
| all‑NVFP4 (V0, superseded) vs itself | 0.59 | 32.0 | device only — same file, GPU vs CPU (noise floor) |
| V1 (GPU) vs all‑NVFP4 (GPU) | 0.50 | 37.5 | the Q8_0 embed swap, same GPU |
| V1 (GPU) vs BF16 (CPU) | 0.52 | 37.0 | quantization + device |
Reading: on this bank the GPU↔CPU kernel difference (τ ≈ 0.5–0.6) is the dominant term, and the NVFP4‑body deltas sit inside that band — the Q8_0 embed swap (V1 vs V0, same GPU) is of the same magnitude as the device noise floor, so it is not a measurable ranking regression here. The clean quantization statement is the same‑device row: Q8_0 preserves 98.3% of document‑pair orderings vs BF16 (Kendall τ 0.97, median rank shift 3 of 269).
License
Apache‑2.0, same as the source model. Quantized for convenience; the original weights and architecture are © Qwen Team — see Qwen/Qwen3-Reranker-0.6B.
Citation
@article{qwen3embedding,
title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
journal={arXiv preprint arXiv:2506.05176},
year={2025}
}