Huihui-Qwen3.8-27B-abliterated-FP8 — v2

⚠️ This is v2. It is quantized from the 2026-08-24 upstream re-release (739e3c5b), in which huihui-ai narrowed the ablation to layers 18–51. The earlier upstream build ablated layers 15–63; the narrower range retains more of the original model's performance. v1 of this repo — the quantization of the older, more heavily ablated weights — has been deleted and is no longer downloadable. If you pulled this repo before 2026-08-24, re-download it.

FP8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with llm-compressor (FP8_DYNAMIC).

huihui-ai publish only BF16 weights and a GGUF build for this model — no FP8 — so this repo fills that gap for vLLM users.

Scheme

256 dense Linear modules are quantized. Everything the upstream Qwen/Qwen3.8-27B-FP8 release leaves alone is left in BF16 here too:

• linear_attn.* — the hybrid Mamba projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj), 48 layers
• the whole vision tower (visual.blocks.*, visual.merger.*)
• embed_tokens, lm_head, and all norms
• the MTP drafter (mtp.*) — kept in BF16 rather than FP8, so speculative decoding still works

Note the upstream FP8 release uses per-tensor weight scales; this one uses per-channel, a finer-grained (and therefore more accurate) scheme at the same size.

Serving with vLLM

vllm serve batsclamp/Huihui-Qwen3.8-27B-abliterated-FP8-v2 \
  --max-model-len 262144 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'

Sampling defaults follow the upstream model card's thinking-mode recommendation (temp 1.0 / top_p 0.95 / top_k 20).

Caveats

• This is an abliterated model: refusal behaviour has been removed upstream. Safety properties are not those of the original Qwen release.
• Quantization was verified structurally and by generation, not by a benchmark suite; no perplexity or eval numbers are claimed.
• With reasoning_effort: xhigh this model family will spend a very large output budget inside the reasoning block — measured 16k tokens / 26 min for one hard question on a GB10. It converges and returns a full answer, but if you cap max_tokens below what it needs you get finish_reason: length and an empty content. This is not specific to this quantization or to abliteration — stock Qwen/Qwen3.8-27B-FP8 behaves identically.