Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4

NVFP4 (W4A4, group 16) quantization of AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 — all credit for the model itself goes to @AEON-7. This repo only carries the quantized weights.

55.6 GB → 20.58 GB. Fits on two 16 GB cards with real KV headroom.

• MTP draft head preserved in bf16 and wired into the index — speculative decoding works (see the note on reasoning effort below).
• bf16 kept for: lm_head, the vision tower, the DeltaNet conv1d, and the MTP head. Everything else is NVFP4 W4A4.
• Built for SM120 / Blackwell with vanilla vLLM v0.22.0 — compressed-tensors is auto-detected, no --quantization flag.
• Quantized GPU-resident (full bf16 dispatched across 7× RTX PRO 2000 Blackwell via device_map=auto): 122 seconds end to end.

Measured (TP=4, 32k ctx, KV fp8, RTX PRO 2000 Blackwell ×4, MTP n=3)

Single-stream prefill: 3,720 tok/s (best 3,888) on an 8k prompt, prefix cache disabled, mean of 3. GPU KV cache at this config: 619,613 tokens.

Sanity-checked after quantization: Japanese self-introduction, arithmetic/primality, and Python code generation all come back clean — no structural damage from the 4-bit pass.

Serve

vllm serve sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4 \
  --trust-remote-code --tensor-parallel-size 4 \
  --max-model-len 32768 --kv-cache-dtype fp8 --reasoning-parser qwen3 \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

On boards without P2P add NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce.

⚠️ Gotchas

• Use reasoning_effort: "medium" for anything long-form. At the default (highest) effort this model can spend its entire token budget inside <think> — the reasoning block grows past 18,000 characters, sometimes degenerating into repeating a single line, and no answer is ever emitted. At medium effort the same prompts finish cleanly every time: I ran a 9-case gate (French and English long-form, temperature 0 and 0.7) in three configurations — NVFP4 with MTP num_speculative_tokens: 6, NVFP4 without MTP, and the original bf16 checkpoint with MTP — and all 27 runs passed with thinking held to 1–3k characters.

  "chat_template_kwargs": {"reasoning_effort": "medium"}
  

  Speculative decoding is fine — an earlier version of this card told you to disable it, which was wrong. MTP at n=6 passes the gate as long as the effort is medium. My apologies for the misdirection, and thanks to the user on Jetson / vLLM 0.27.1 who reported the looping.

• The 15 mtp.* modules are listed in quantization_config.ignore — do not remove them. If vLLM treats the bf16 MTP head as NVFP4 the draft breaks silently: 0% acceptance, slower than no MTP at all.

• W4A16 (NVFP4A16) does not serve on this architecture in vLLM 0.22 (gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64). W4A4 only.

• Long single-file code generation drops a closing paren roughly 1–2 times in 14 regardless of sampling temperature (measured with a JS parser on the base Qwen3.8-27B). Put a syntax check in the loop rather than tuning temperature.

Recipe

llm-compressor NVFP4 (W4A4, group 16), targets: [Linear], ignore: [lm_head, re:.*visual.*, re:.*conv1d.*, re:.*mtp.*], 32 calibration samples × 8192 tok from neuralmagic/calibration. MTP tensors grafted back in bf16 after saving and appended to quantization_config.ignore.

🙏 @AEON-7 for the model, Qwen team for the base, vLLM & llm-compressor teams for the tooling.