Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF

GGUF conversion of Qwen/Qwen3.8-27B, quantized to NVFP4 in the MLP with NVIDIA ModelOpt, keeping the native MTP layer for speculative decoding and a BF16 vision projector for image input.

This is the accuracy-oriented Q8attn variant: the attention and linear-attention (DeltaNet) projections are stored at Q8_0 (≈ the FP8 precision they were quantized to), rather than being dropped to Q4_K. That keeps accuracy up at long context, where 4-bit attention / DeltaNet projections hurt most. Benchmarked on an RTX 5090.

Highlights

• NVFP4 MLP (self-quantized): 193 NVFP4 MLP tensors, produced with NVIDIA ModelOpt from the full-precision Qwen/Qwen3.8-27B.
• Q8 attention: attention (q/k/v/o) and linear-attention / DeltaNet projections (attn_qkv, attn_gate, ssm_out, ssm_alpha, ssm_beta) are stored as Q8_0, for better accuracy than a 4-bit-attention build.
• MTP included: the GGUF keeps the native MTP layer for draft-mtp speculative decoding, roughly doubling generation throughput on this hardware.
• Vision supported: ships a BF16 mmproj projector for image input (Qwen3-VL vision path).
• RTX 5090 tested: MTP verified with a draft-n sweep; ~120–130 tok/s generation with MTP.

Provenance

Files

llama.cpp example

# Text-only
llama-server \
  -m Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -c 213056 \
  -ngl 999

# With vision
llama-server \
  -m Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf \
  --mmproj mmproj-Qwen3.8-27B-NVFP4-BF16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -c 163840 \
  -ngl 999

The context lengths above are what this repo's author runs on a 32 GB RTX 5090: -c 213056 for text-only and -c 163840 when the vision projector is loaded (the mmproj plus its image tokens take extra VRAM, so a smaller context leaves headroom).

A note on MTP depth. A draft-n sweep on this model (below) shows generation throughput plateaus around --spec-draft-n-max 2–3. d=2 gives the highest draft-acceptance and the most stable throughput; d=3 reaches a slightly higher peak on code-like workloads. Values of d≥4 gave no reliable speedup (acceptance collapses). d=2 or d=3 are both good choices.

Benchmarks

Measured on RTX 5090, -ngl 999, c=32768, KV q8_0, generation of 512 tokens (ignore_eos), draft-acceptance and generation t/s read from the server's own timings (ground truth). n = --spec-draft-n-max; n0 = autoregressive baseline (no MTP).

MTP roughly doubles generation throughput over the non-speculative baseline (65 → ~124 tok/s). Throughput plateaus at n2; beyond n3 the draft-acceptance rate collapses and there is no reliable gain. (End-to-end sanity check at c=213056: MTP active, ~78–86% acceptance, ~121–129 tok/s.)

Model details

Tensor composition of the main GGUF

The conversion keeps the NVFP4 MLP tensors instead of requantizing all weights to K-quants, and keeps attention / DeltaNet projections at Q8_0 rather than Q4_K.

License and attribution

The upstream model is licensed under Apache-2.0:

• Qwen/Qwen3.8-27B

This repository redistributes a GGUF-format, NVFP4-quantized conversion of Qwen3.8-27B under the same Apache-2.0 license. Please also follow the terms and usage guidance of the upstream model card.

Acknowledgements

Thanks to the Qwen team for the base model, NVIDIA for the ModelOpt NVFP4 quantization toolkit, and the llama.cpp project for GGUF, NVFP4, MTP, and multimodal runtime support.