Qwen3.8-27B-NVFP4-AWQ-GPTQ
Mixed-precision NVFP4 quantization of Qwen/Qwen3.8-27B,
built with llm-compressor using
AWQ activation-aware scaling followed by GPTQ, with an imatrix_mse observer.
24.7 GB. At the same size as a plain NVFP4 build, it cuts high-confidence damage by roughly a third, and it is the most accurate NVFP4 checkpoint of this model we have measured.
Recipe
| component | precision |
|---|---|
mlp.{gate,up,down}_proj, layers 0–55 | NVFP4 (4-bit, group-16, FP8-e4m3 scales → 4.5 effective bits) |
mlp.{gate,up,down}_proj, layers 56–63 | FP8 e4m3 (dynamic) |
self_attn.{q,k,v,o}_proj | FP8 e4m3 (dynamic) |
linear_attn.{in_proj_qkv,in_proj_z,out_proj} (GDN) | FP8 e4m3 (dynamic) |
lm_head, embed_tokens, all norms, GDN state params, vision tower | BF16 |
Two passes, in order:
- AWQ — per-input-channel scaling on
post_attention_layernorm → {gate_proj, up_proj}andup_proj → down_proj. Gate and up share one input, so the reciprocal scale folds into the norm weights: the accuracy gain costs zero bytes and zero throughput. The scales merge into weights entirely, so unlike rotation-based methods (QuIP/SpinQuant) this checkpoint still runs under tensor parallelism. - GPTQ on every quantized module (
actorder="static",dampening_frac=0.01).
Calibration: 1024 sequences × 1024 tokens of a balanced Nemotron-v2 blend (25% code, 25% math, 20% STEM, 20% chat, 10% multilingual).
lm_head and embed_tokens are left in BF16 — matching Qwen's own official FP8 release,
which does the same.
Benchmarks
Measured against the BF16 base model on 142,727 tokens of self-distilled thinking-mode output, plus 200 free greedy generations. vLLM 0.27.1, TP=2, 2×B300.
| checkpoint | size ↓ | top-1 ↑ | near-tie ↓ | moderate ↓ | confident ↓ | certain ↓ | divmed ↑ | tok/s ↑ |
|---|---|---|---|---|---|---|---|---|
Qwen/Qwen3.8-27B-FP8 (8-bit ref) | 30.9 GB | 96.15% | 22.70% | 3.48% | 1.45% | 0.08% | 47 | 8711 |
| this model (NVFP4+AWQ) | 24.7 GB | 93.44% | 33.86% | 7.74% | 2.69% | 0.19% | 29 | 10680 |
RadixArk/Qwen3.8-27B-NVFP4 | 21.9 GB | 90.23% | 43.80% | 14.49% | 3.29% | 0.70% | 11 | 11436 |
unsloth/Qwen3.8-27B-NVFP4 | 23.4 GB | 91.75% | 40.12% | 10.32% | 3.91% | 0.25% | 19 | 11069 |
Bold marks the best value in each column among the FP4 checkpoints; the FP8 row is a reference at a different precision and size class, so it is excluded from the comparison. All sizes are on-disk tensor bytes and include the ~0.85 GB BF16 MTP head, which every checkpoint in this table ships. Subtract ~0.85 GB for a no-MTP comparison.
Columns. top-1 is raw argmax agreement with BF16. The four bucket columns are
disagreement rates, split by how confident the base model was at that position
(top1−top2 logprob margin): near-tie <0.5, moderate 0.5–2, confident 2–5,
certain >5. Only confident and certain represent real damage — a flip where
the base model itself was nearly tied is numerical noise, not a quality loss.
divmed is the median token index at which free greedy generation first diverges
from BF16 (higher is better).
Perplexity is deliberately excluded. On this comparison it is anti-correlated with
quality — the checkpoint with the best perplexity (RadixArk, −1.75%) has the worst
certain-bucket damage of any arm measured (0.70%, 3.7× this model's). Do not rank
FP4 checkpoints of this model by perplexity.
In an internal ablation, removing the AWQ pass and keeping everything else identical
raises confident damage from 2.69% to 3.97% — so AWQ closes about half of the
gap to FP8, at no size or speed cost.
Usage
from vllm import LLM
llm = LLM("selimaktas/Qwen3.8-27B-NVFP4-AWQ-GPTQ", tensor_parallel_size=2)
Requires a Blackwell-class GPU for native NVFP4, and vLLM with compressed-tensors.
Speculative decoding (MTP)
The model's MTP (multi-token prediction) head is included, in BF16, and works with
vLLM's mtp speculative decoding:
from vllm import LLM
llm = LLM("TelperionAI/Qwen3.8-27B-NVFP4-AWQ-GPTQ", tensor_parallel_size=2,
speculative_config={"method": "mtp", "num_speculative_tokens": 2})
Qwen3_5ForConditionalGeneration does not carry mtp.* in its state dict, so
llm-compressor never sees it and it is silently dropped, even though config.json still
declares mtp_num_hidden_layers: 1. It is grafted back in here from the base checkpoint
and excluded from quantization (re:.*mtp.* in
quantization_config.ignore; without that exclusion the quantization target regexes also
match mtp.layers.0.mlp.* and vLLM fails to load). Draft quality drives acceptance rate,
so it is kept at full precision rather than quantized.
Acceptance rate has not been measured; the head is verified to load and generate.
Limitations
- Single evaluation corpus. All numbers come from one self-distilled corpus. The margins over the public NVFP4 checkpoints are large and statistically solid, but the comparison has not been replicated on a second distribution.
- Vision tower is untouched (BF16); this was evaluated as a text model.
KV-cache quantization (calibrated scales included)
This checkpoint carries per-layer k_scale / v_scale in
model-kv-scales.safetensors, with kv_cache_scheme under
quantization_config — the on-disk contract llm-compressor emits and vLLM reads
in BaseKVCacheMethod. Without them vLLM falls back to a scale of 1.0.
--kv-cache-dtype fp8 # 32 KiB/token, from 64 KiB at fp16
Provenance: these scales were measured on Qwen3.8-27B-NVFP4-AWQ-AutoRound, a different quantization of the same base model, over a 2,353-document / 3.7M-token corpus — the size at which the post-RoPE V absmax stops moving. K/V range is a property of the model rather than of the weight format (per-layer amax agrees to a median ratio of 1.0002 across our trellis / FP8 / INT4 / NVFP4 builds), so they transfer. They were not measured on this checkpoint specifically; cross-build agreement is excellent at the median but ranges 0.88–1.36 in the tails.