Qwen3.8-Flash-Next MTP draft head (GGUF, standard quant types)
The multi-token-prediction head that ships inside
Qwen/Qwen3.8-Flash-Next: one full
qwen4exp block (attention + 512-expert MoE + hyper-connections, 4B parameters), trained
jointly with the model. The upstream converter (PR 27742) drops it, so the popular GGUFs
have none. This one is exported straight from the bf16 checkpoint and quantized with
plain llama-quantize, so it runs on stock llama.cpp kernels (any backend), not a fork's
quant types.
This is a draft head, not a model. Use it as -md next to any Flash-Next GGUF
target (tested with unsloth's UD-Q4_K_XL and UD-IQ4_XS).
| file | size | what |
|---|---|---|
Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf | 2.5 GB | the one to use: Q4_K_M with the LM head and token embeddings at Q4_K (reading the draft's LM head is most of each draft step; and a draft quantized like its target agrees with it more often than a more precise one -- a Q8_0 head measured worse) |
Qwen3.8-Flash-Next-MTP-BF16.gguf | 7.8 GB | unquantized export, to requantize as you like |
Needs a patched llama.cpp (for now)
qwen4exp is not upstream yet and neither is its MTP graph. Two options:
- llama.cpp
7584430716ee(tag b10612) + PR 27742 (model: add Qwen3.8-Flash-Next, pinned atunslothai/llama.cpp@bea3b12d) +patches/qwen4exp-mtp-draft-head.patchfrom this repo. The patch is the draft-head graph (JJJYmmm's PR 27739, reconciled by LaurentZuijdwijk) and the converter's--mtpexport, plus one fix (an MTP-only export emits no PLE keys). Verified to apply and build on exactly that tree. - Or build LaurentZuijdwijk/llama.cpp
branch
vulkan/qwen4exp-rocmfpx, which carries the same two commits (b98aa9847, ae96a0dc0) on a Vulkan backend tuned for Strix Halo.
Set LLAMA_ATTN_ROT_DISABLE=1: upstream's quantized-KV activation rotation (#21038) is
not supported by the qwen4exp attention path and the server aborts at load without it.
Run
LLAMA_ATTN_ROT_DISABLE=1 llama-server \
-m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
-md Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf -ngld 999 \
--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 \
-ngl 999 -fa on -ctk q8_0 -ctv q8_0 -c 262144
p-min 0.75 stops drafting when the head is unsure; that matters more than depth. Deeper
fixed drafting (n-max 8, p-min 0.3) measured slower, and so did adaptive drafting: this
head carries its own MoE, so every drafted token is a real forward pass.
The target's 28.8 GB n-gram table (per_layer_token_embd) is the model's problem, not the
head's: upstream keeps it in host memory (-ot per_layer_token_embd=CPU), Laurent's fork
splits it per head so it fits a Vulkan device. Our numbers below were taken with a patch
that leaves it on disk and reads rows per batch; it changes memory, not speed.
Attached to unsloth's UD-Q4_K_XL (no -md needed)
unsloth-UD-Q4_K_XL-mtp-shards/ turns
unsloth/Qwen3.8-Flash-Next-GGUF's
UD-Q4_K_XL into a model that carries its own draft head, the way Qwen3.5/3.6-MoE GGUFs do
(nextn_predict_layers = 1, the head as blk.48.*). Two files, 1.9 GB, no re-download of
the 111 GB:
- keep unsloth's shards 2, 3 and 4 and rename them
-0000N-of-00005.gguf - drop in
…-00001-of-00005.gguf(their shard 1 with the metadata changed:block_count 49,nextn_predict_layers 1,compress_ratiosextended,split.count 5) and…-00005-of-00005.gguf(the 29 head tensors, the same bytes as the Q4_K_M head above) - run with
--spec-type draft-mtpand no-md; the draft context is created against the target and shares its memory (~0.5 GB less than the separate head)
Made with patches/merge-mtp-shard.py, which does the same to
any split qwen4exp GGUF given a head file. Measured: within a few percent of the -md form
(34.5 vs 36.7 t/s on code, same on prose) -- the convenience is the point, not speed.
Measured (AMD Strix Halo / Radeon 8060S, 128 GB unified, temperature 0, 300 tokens)
| backend, target | bare | + MTP | acceptance (code / prose) |
|---|---|---|---|
| ROCm, UD-Q4_K_XL | 20.3 t/s | 35.8 code / 22.6 prose | 0.90 / 0.74 |
| ROCm, UD-IQ4_XS | 18.0 / 18.6 | 32.8 / 22.1 | 0.84 / 0.68 |
| Vulkan (RADV, Laurent's fork), UD-IQ4_XS | 24.2 / 24.3 | 37.2 code / 30.3 prose | 0.88 / 0.82 |
Where the remaining time goes on this GPU, profiled: a target pass is ~47 ms for one token and only ~4.4 ms per extra verified token, a head step ~3.4 ms, and about a third of each pass is kernel-launch gaps (the target graph is ~8,000 nodes). Vulkan launches cheaper, which is the whole of its lead.
How it was made
# the 31 bf16 shards that hold mtp.* plus embed_tokens / lm_head / the PLE constants (58 GB)
python convert_hf_to_gguf.py --mtp --outtype bf16 --outfile Qwen3.8-Flash-Next-MTP-BF16.gguf src/
llama-quantize --output-tensor-type Q4_K --token-embedding-type Q4_K \
Qwen3.8-Flash-Next-MTP-BF16.gguf Qwen3.8-Flash-Next-MTP-Q4_K_M.gguf Q4_K_M
34 tensors: blk.48.* (attention, indexer, MoE, hyper-connections), blk.48.nextn.{eh_proj,enorm,hnorm},
token_embd, output, output_hc_*; nextn_predict_layers = 1, block_count = 49.
Credits
qwen4exp support: Daniel Han (PR 27742). MTP graph: JJJYmmm (PR 27739), reconciled and
the converter export by LaurentZuijdwijk.
The "match the target's quant" observation: agentionai. Base model: the Qwen team,
Qwen Community License 1.0 (included).