Qwen3.8-27B Selective NVFP4 MLP GGUFs
Two selective NVFP4 conversions of Qwen/Qwen3.8-27B for llama.cpp: an accuracy-oriented BF16-attention build and a smaller Q8-attention build.
Both were independently calibrated from the official Qwen BF16 checkpoint using NVIDIA Model Optimizer. In both variants, the 64-layer backbone MLP uses W4A16 NVFP4 while embeddings, the output head, MTP weights, and vision components remain at higher precision. The variants differ in how their attention weights are stored.
Files
| File | Purpose | Size |
|---|---|---|
Qwen3.8-27B-NVFP4-W4A16-MLP-BF16attn-MTP.gguf | Accuracy-oriented variant with BF16/F32 attention and embedded MTP/NextN weights | 30.06 GB |
Qwen3.8-27B-NVFP4-MLP-Q8attn-BF16head-MTP.gguf | Compact variant with Q8_0 attention, BF16/F32 output head, and embedded MTP/NextN weights | 23.29 GB |
mmproj-Qwen3.8-27B-BF16.gguf | BF16 vision projector for image input | 0.93 GB |
Qwen3.8-27B-DFlash2-Q4_K_M.gguf | Qwen3.8-27B DFlash 2 Q4_K_M drafter | 1.14 GB |
qwen38_27b_nvfp4_w4a16_mlp_only.yaml | ModelOpt recipe for the BF16-attention variant | — |
qwen38_27b_nvfp4_mlp_fp8_attn_w4a16.yaml | ModelOpt recipe for the compact variant; calibrated FP8 attention weights are stored as Q8_0 in GGUF | — |
Either model GGUF works by itself for text. Add the same mmproj file for image input with either variant.
Choosing a variant
| Variant | Best for | Trade-off |
|---|---|---|
| BF16 attention, 30.06 GB | Maximum retained attention precision | Higher VRAM and storage use |
| Q8_0 attention, 23.29 GB | Lower VRAM and faster model loading | Attention weights use 8-bit block quantization |
Quantization layout
In both files, W4A16 MLP means the large feed-forward weights use 4-bit NVFP4 while their inference activations remain 16-bit. The gate_proj, up_proj, and down_proj weights were quantized in each of the 64 backbone layers, producing 192 NVFP4 tensors.
The 30.06 GB variant retains the attention paths at BF16/F32. For the 23.29 GB variant, softmax-attention and the large linear-attention projections were calibrated as FP8 by Model Optimizer and stored as Q8_0 during GGUF conversion. FP8 activation scales are not used by llama.cpp; Q8attn describes the final GGUF weight storage.
| Component | BF16-attention variant | Q8-attention variant |
|---|---|---|
Backbone MLP gate_proj, up_proj, down_proj | NVFP4 W4A16 | NVFP4 W4A16 |
| Self-attention Q/K/V/O weights | BF16/F32 | Q8_0 |
| Large linear-attention projection weights | BF16/F32 | Q8_0 |
| Smaller linear-attention state/conv components | BF16/F32 | BF16/F32 |
Token embeddings and lm_head | BF16/F32 | BF16/F32 |
| MTP/NextN weights | BF16/F32, embedded | BF16/F32, embedded |
| Vision tower/projector | Shared BF16 mmproj | Shared BF16 mmproj |
| KV cache | Selected at runtime | Selected at runtime |
Running with embedded MTP
A recent CUDA build of llama.cpp with native NVFP4 and Qwen3.8 support is required. The following starts the built-in web UI and uses the MTP/NextN weights already contained in the selected model GGUF:
llama-server \
-m Qwen3.8-27B-NVFP4-W4A16-MLP-BF16attn-MTP.gguf \
--mmproj mmproj-Qwen3.8-27B-BF16.gguf \
--ctx-size 32768 \
--flash-attn on \
--n-gpu-layers -1 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--spec-draft-p-min 0.75 \
--reasoning on \
--reasoning-effort xhigh \
--reasoning-preserve \
--jinja \
--ui
For the compact variant, replace the model filename after -m with Qwen3.8-27B-NVFP4-MLP-Q8attn-BF16head-MTP.gguf; all other options remain the same. Remove --mmproj for text-only use. Increase context size according to available VRAM.
Thinking controls
Qwen3.8 uses thinking mode by default. Its default reasoning effort is xhigh, and preserved thinking is enabled by default. The launch command above makes those defaults explicit.
| llama.cpp flag | Effect |
|---|---|
--reasoning on | Enable thinking |
--reasoning off | Disable thinking for direct responses |
--reasoning auto | Let the model's chat template select the mode |
--reasoning-effort xhigh | Thorough reasoning; Qwen3.8 default |
--reasoning-effort medium | Balance reasoning depth and speed |
--reasoning-effort low | Prefer shorter, faster reasoning |
--reasoning-budget N | Limit thinking to N tokens; -1 is unrestricted |
--reasoning-preserve | Retain reasoning context across conversation history |
--no-reasoning-preserve | Do not retain earlier reasoning blocks |
--reasoning-format only controls how thinking is returned by the API. Its choices include none, deepseek, and deepseek-legacy; the default auto mode normally selects the appropriate parser.
Recommended DFlash 2 acceleration
These GGUFs can also be used as targets for the separate Qwen3.8-27B DFlash 2 Q4_K_M drafter. The drafter is not required to run either model.
DFlash 2 requires a compatible llama.cpp build; this combination was tested using DFlash 2 PR #27342.
llama-server \
-m Qwen3.8-27B-NVFP4-W4A16-MLP-BF16attn-MTP.gguf \
-md Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
--mmproj mmproj-Qwen3.8-27B-BF16.gguf \
--ctx-size 32768 \
--flash-attn on \
--n-gpu-layers -1 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--spec-type draft-dflash \
--spec-draft-n-max 7 \
--spec-draft-ngl all \
--cache-type-k-draft q8_0 \
--cache-type-v-draft q8_0 \
--reasoning on \
--reasoning-effort xhigh \
--reasoning-preserve \
--jinja \
--ui
One local CUDA run with the BF16-attention target at 32,768 context used the default xhigh thinking effort, the Q4_K_M drafter, and Q8_0 target/draft KV caches. It generated 4,628 tokens at 103.18 tokens/s, with 51.68% draft acceptance (3,625 of 7,014 draft tokens) and a mean accepted length of 4.62. This is a single-system result, not a universal performance claim; speed and acceptance vary with hardware, prompt, sampling, context, and runtime build.
License and attribution
The source model is released under Apache-2.0. The upstream license is included as LICENSE. Use of these files remains subject to the source model's terms and applicable law.
- Source model: Qwen/Qwen3.8-27B
- NVIDIA Model Optimizer: NVIDIA/Model-Optimizer
- GGUF runtime and converter: ggml-org/llama.cpp