Qwen3.8-27B DFlash2 draft model, ROCmFP4-FAST GGUF
A 4.25 bpw ROCmFP4 requantisation of z-lab/Qwen3.8-27B-DFlash2, for use as a speculative decoding sidecar with a Qwen3.8-27B target.
Measured on an AMD Strix Halo (Radeon 8060S), paired with julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF: 65.6 t/s on structured output, 4.7x bare decode.
Two things to know before you download
1. This will not load in stock llama.cpp. Q4_0_ROCMFP4_FAST is GGUF file type 103, which
exists only in the fork linked below. Upstream llama.cpp fails with
failed to load model before it reaches the GPU. Get a prebuilt binary here:
2. This is a draft model, not a standalone one. It has no full graph of its own. Loading it by
itself fails with failed to create context, which is expected. It must be passed with -md
alongside a target model.
Usage
llama-server \
-m Qwen3.8-27B-ROCmFP4-FAST.gguf \
-md Qwen3.8-27B-DFlash2-Q4_0_ROCMFP4_FAST.gguf \
--spec-type draft-dflash --spec-draft-adaptive \
--spec-draft-n-min 3 --spec-draft-n-max 7 --spec-draft-ngl 99 \
-ngl 999 -fa on -b 2048 -ub 512 -c 32768
--spec-draft-adaptive sizes the draft from measured acceptance instead of a fixed n. It is
what makes n-max 7 safe here: at a fixed n=7 acceptance collapses to 18 % and throughput
drops to 20.2 t/s, while adaptive holds 96 % acceptance and reaches 65.6.
| Qwen3.8-27B, FP4 target + this sidecar, greedy, 300 tokens | structured output | prose |
|---|---|---|
| bare decode | 14.0 t/s | 14.1 t/s |
| fixed draft n=3 | 41.6 | 25.4 |
| fixed draft n=7 | 20.2 | 24.8 |
adaptive, n_max 7 n_min 3 | 65.6 t/s - 4.7x | 26.1 |
Speculative decoding raises throughput, not quality: the target model verifies every token, so output matches what the target would have produced on its own.
Requirements
- The fork above, or any build with the ROCmFPx quant types
- A Vulkan 1.3 driver. Measured on Mesa RADV 26.0.8 on gfx1151.
- The FP4 path is tuned for AMD Strix Halo. It should run anywhere the fork builds, but the numbers above are specific to this hardware.
A note on requantising
Do not requantise a DFlash2 sidecar to Q8_0_ROCMFPX expecting parity with Q8_0. At identical
bits per weight, ours scored 53.5 % acceptance against z-lab's 60.2 %. The cause is the block
scale, not the codes: Q8_0 stores an fp16 scale, ROCmFPx stores a UE4M3 byte. At 8 bits per
weight the coarse scale is what costs you. FP4 is a different trade and is the one measured here.
Credits
- z-lab/Qwen3.8-27B-DFlash2, the source model (Apache-2.0)
- ciru-ai/ROCmFPX for the FPx quant formats
- DFlash2 support in llama.cpp, PR #27342 by Jian Chen
- ggml-org/llama.cpp
Quantized and published by Agention.
Licensed Apache-2.0, inherited from the base model.