Huihui-Qwen3.8-27B-abliterated — NVFP8
Block-16, two-level-scaled FP8 E4M3. 55.6 GB → 32.8 GB (−41%).
Stock vLLM cannot load this
NVFP8 is not an NVIDIA-defined format. NVIDIA's microscaling family is NVFP4
(block-16, E4M3 scales) at 4 bits and MXFP8 (block-32, E8M0 scales) at 8 bits —
there is no vendor NVFP8. vLLM dispatches kernels by matching quantization
args: _is_nvfp4_format requires num_bits == 4, _is_mxfp8 requires
group_size == 32. This checkpoint's signature — 8-bit, tensor_group,
group_size=16 — matches nothing, and loading raises NotImplementedError.
The nvfp8/ package in this repo supplies the Triton kernel and the dispatch
hook that make it loadable.
Usage
The runtime ships in this repo, so download it as a directory rather than letting vLLM stream the weights into the HF cache — you need a local path to install the plugin from.
hf download trailio/Huihui-Qwen3.8-27B-abliterated-NVFP8 --local-dir ./qwen-nvfp8
cd qwen-nvfp8
pip install -e . # registers a vllm.general_plugins entry point
from vllm import LLM
llm = LLM("./qwen-nvfp8", trust_remote_code=True, max_model_len=8192)
print(llm.generate(["The capital of France is"])[0].outputs[0].text)
pip install -e . has to run from that downloaded directory — it is what
registers the entry point. Passing the repo id straight to LLM(...) without
installing gets you the NotImplementedError described above, because the
weights land in the HF cache and the plugin is never installed.
The entry point is required rather than a convenience: vLLM forces the spawn
start method once CUDA is initialized, so calling register() by hand in the
parent process never reaches EngineCore. Installation is what makes
registration happen in every engine and worker process.
Needs ~31 GB of disk for the download and ~35 GB of VRAM at runtime (weights plus activations), before KV cache.
Requires sm_89+ (Ada or newer). Validated on RTX PRO 6000 Blackwell (sm_120, 96 GB) with vLLM 0.27.1.
Format
weight [N, K] float8_e4m3fn
weight_scale [N, K/16] float8_e4m3fn one per 16 input elements
weight_global_scale per-shard float32
Weight-only (W8A16); activations stay bf16.
The global scale is per-shard rather than per-tensor because vLLM fuses
gate_proj+up_proj into gate_up_proj and q/k/v_proj into qkv_proj.
Those tensors were quantized independently and carry different global scales,
so the value is expanded per-output-row at load time. Collapsing it to a single
scalar mis-scales one shard by the ratio between them.
Group scales are rounded up to the next representable e4m3 value, not to nearest. Round-to-nearest lets a scale land below its target, pushing the group's largest element past 448 so it clips — measured at 3.4% of all elements. Rounding up drives that to zero.
What was quantized
400 Linear layers, 24.33B of 27.78B params (87.6%). Left in BF16:
| why | |
|---|---|
visual.* | 27-block ViT + merger — abliteration left vision untouched |
mtp.* | multi-token-prediction head; accept rates are sensitive to its logits |
lm_head, embed_tokens | 1.27B params each; vocab projections are where 8-bit bites |
linear_attn.in_proj_a/b | [48, 5120] DeltaNet decay/gate — 245K params each, and error compounds through the SSM recurrence instead of averaging out |
A_log, dt_bias, conv1d | SSM state; config pins mamba_ssm_dtype=float32 |
All quantized K-dims (5120, 6144, 17408) divide by 16, so groups tile with no padding.
Accuracy
Weight round-trip error, identical tensors:
| scheme | rel. error |
|---|---|
| NVFP8 (block-16) | 0.0252 |
| FP8 per-channel | 0.0265 |
| FP8 per-tensor | 0.0265 |
On outlier-heavy weights: 0.0213 vs 0.0249 per-channel. 0.025 is the E4M3 floor — 3 mantissa bits — not a property of the scaling scheme.
Triton kernel error against an fp32 reference is ≤3.7e-3 across every layer shape at M=1 through M=2048, roughly an order of magnitude below the quantization error itself.
PTQ with no calibration data: weight-only NVFP8 derives its scales from the weights, so there is no dataset dependence and nothing to overfit.
Known issues
On containers with CUDA < 12.9, FlashInfer's JIT cannot resolve sm_120 and
fails with FlashInfer requires GPUs with sm75 or higher. Set
VLLM_USE_FLASHINFER_SAMPLER=0 or use a CUDA ≥ 12.9 image. Unrelated to
quantization — it affects the sampler, not the model.
Provenance
Quantized from huihui-ai/Huihui-Qwen3.8-27B-abliterated, itself an abliterated derivative of Qwen/Qwen3.8-27B.
Abliteration removes refusal behavior; the base model's reduced-filtering caveats apply here unchanged. Quantization preserves behavior — it neither adds nor removes safety properties.