Qwen3.8-Flash-Next GGUFs — provenance-verified quants for Strix Halo
Four files:
| file | size | what it is |
|---|---|---|
Qwen3.8-Flash-Next-IQ4_XS-PLE.gguf | 91 GiB | recommended daily driver — iq4_xs trunk with the 27G PLE n-gram table at iq4_nl |
Qwen3.8-Flash-Next-IQ4_XS.gguf | 116 GiB | static reference quant, PLE at q8_0 |
Qwen3.8-Flash-Next-IQ4_XS-M2.gguf | 115 GiB | imatrix-calibrated quant, PLE table at q8_0 — best measured perplexity |
mtp-Qwen3.8-Flash-Next-Q8_0.gguf | 3.9 GiB | MTP draft sidecar for nathanw1014-lineage engines (fork-specific — will NOT load on apepojken/mainline) |
M2 — the imatrix quant
second-generation quant: same trunk type (iq4_xs), but calibrated with a
926-entry imatrix (1,024 chunks) via the ROCmFPX banded quantizer, and the
51B PLE lookup table left at q8_0 (no --tensor-type cut). 5.56 bpw,
115 giB — 24 giB bigger than the 91g PLE file.
perplexity (wiki.test.raw, ctx 2048, 145 chunks):
| quant | PPL |
|---|---|
| M2 (imatrix, PLE q8_0) | 4.2809 ±0.025 |
| PLE 91g | 4.2932 ±0.025 (statistically tied, <0.5σ) |
| static 116g | 4.5221 ±0.026 (~9σ worse) |
speed profile is honest-mixed (single runs, nathanw1014 vulkan engine, q8_0 kv, temp 0):
| depth | plain tg | mtp tg |
|---|---|---|
| 8k | 23.8 (≈ PLE 23.8) | 25.1 (PLE 33.5 — M2 slower) |
| 32k | 19.0 (≈ PLE 19.0) | 29.1 (best of the three) |
| 128k | — | 13.3 (PLE 13.5 — tied) |
pick M2 when you want the best measured quality and don't mind the 24 giB: at ≤32k plain it matches PLE, and at 32k MTP it measured fastest. for shallow-depth MTP speed take the 91g PLE; for deep 128k+ MTP the static 116g had a small in-sweep edge (17.4 vs 13.3/13.5 — within the same-config spread, n=1 caveat).
provenance
every quant descends from an F16 that was byte-verified against the official
Qwen/Qwen3.8-Flash-Next-FP8 checkpoint: hyper-connection norms folded to
(1 + w) (97/97 tensors — the converter bug that produces deterministic garbage
is fixed in our pipeline), PLE fp8 scale applied, expert stacking identity
probed 512x3, GDN v-head reorder checked. details:
https://github.com/julianmb/haloq38flash
the PLE cut (what makes the 91G special)
the 51B-parameter n-gram lookup table was moved from q8_0 (54G) to iq4_nl (27G) via --tensor-type. hash-gathered lookup rows tolerate the precision drop — verified by smoke and full benchmark, no degradation observed:
| depth | static 116G plain/mtp t/s | PLE 91G plain/mtp t/s |
|---|---|---|
| 0 | 29.2 / 48.4 | 29.9 / 53.1 |
| 8k | 22.9 / 42.8 | 24.1 / 56.4 |
| 32k | 19.5 / 29.5 | 20.1 / 30.2 |
prefill at 32k: 384 → 397 t/s. no collapse at depth. engine: nathanw1014 strix-halo-vulkan (ad914eb), vulkan/radv, q8_0 KV, -ub 2048, temp 0.
fork compatibility caveat (important)
the iq4_nl PLE rows assert in SOME forks: engines that feed gathered PLE rows directly as mul_mat B operands without dequantizing (apepojken qwen4exp-spec-mtp) abort at ggml-vulkan.cpp:7794 (b_type must be F32/F16/Q8_1). verified working on nathanw1014 strix-halo-vulkan. if your engine asserts on load or first token, use the 116G static file instead. M2 keeps the PLE table at q8_0 and has no such assert exposure.
provenance note
the same --tensor-type cut applied to unverified-source quants will NOT fix a broken converter (hc norms, PLE scale) — garbage in, garbage out. ours is built from a fixed, audited pipeline.
128k & 256k context benchmarks (SSD-PLE)
the depth story is not monotonic. measured on the same engine (nathanw1014 vulkan, q8_0 kv, temp 0):
| depth | static 116g plain/mtp t/s | PLE 91g plain/mtp t/s |
|---|---|---|
| 0 | 29.2 / 48.4 | 29.1 / 48.1 |
| 8k | 22.9 / 42.8 | 21.8 / 27.9 (peak: 56.4) |
| 32k | 19.5 / 29.5 | 18.5 / 25.5 (peak: 30.2) |
| 128k | 10.8 / 26.9 | 8.9 / 11.8 (peak: 18.6) |
| 256k | 6.2 / — | 6.0 / 15.2 (SSD-PLE tuned) |
- 256k context unlocked with SSD-PLE: With
-lm mmap --tensor-read-lazy on, the 27 GB PLE table is kept on NVMe SSD, keeping active RAM under 95 GB and providing ~30 GB of free headroom at full 256k context. Combined with-tb 16and-ub 1024 -b 2048 -t 4, this achieves 15.2 t/s MTP generation and 179-191.5 t/s prefill at 256k with zero swap thrashing. - Hardware boundary: Do not use
-ub 2048at 256k context; it triggers Vulkan queue submission timeouts (ErrorDeviceLost).-ub 1024is the tested maximum. - 128k reversal: At 128k specifically under MTP, the static 116g quant has higher draft acceptance than the 91g PLE quant (26.9 vs 18.6 t/s).
- Practical: Use the 91g PLE file for daily chat, coding, and full 256k long context (with SSD-PLE flags). Use the 116g static file if you specifically need highest MTP speed at 128k.
run it (strix halo, 128 GB unified memory)
full methodology, receipts, and the benchmark record: https://github.com/julianmb/haloq38flash
docker (one-liner; image default serves the 91g PLE on :8080):
git clone https://github.com/julianmb/haloq38flash && cd haloq38flash
docker compose up --build
point it at M2 with the MTP sidecar and the warm-turn cache:
docker compose run qwen38-flash-next /app/llama-server \
-m /models/Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
-md /models/mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
--cache-ram 8192 --ctx-checkpoints 32 \
-c 32768 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4
or raw llama.cpp (nathanw1014 strix-halo-vulkan lineage engines):
llama-server -m Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
-md mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
-ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4 -c 32768
perf notes: -t 16 lifts prefill up to +43% at 128k (decode indifferent);
the --cache-ram/--ctx-checkpoints warm-turn flags make repeated
context nearly free (measured 438 s → 0.68 s at 128k, 994 s → 0.74 s at
256k — 640x/1351x). swap M2's filename for the other quants; same flags.
license
qwen community license 1.0 (distribution permitted with notice; maas restrictions apply). base model: Qwen/Qwen3.8-Flash-Next.