Qwen3.8-Flash-Next AP GGUF
Agention Precision (AP) quants of Qwen/Qwen3.8-Flash-Next. Mainline llama.cpp compatible — standard quant types only, no fork required.
Updated 2026-10-13:
AP-Q5_K_M. Same file name, size and tensor types, better weights from an improved encode: KL divergence 0.0771 and top-1 agreement 86.95 % on held-out text, 0.0181 on code: level with unsloth's UD-Q4_K_XL on held-out text and ahead on code, in the same VRAM. Download the file again.Updated 2026-10-12:
AP-Q4_K_XL. KL divergence 0.0886 and top-1 agreement 86.14 % on held-out text, 0.0206 on code. 0 loops in a 45-run loop stress test.
Each tier varies the quant type per tensor group rather than using one type throughout, to get the most accuracy per gigabyte of VRAM.
Tiers
| tier | VRAM | n-gram table (host) | download size | eff. bpw | experts (gate/up) | KL div | top-1 agree |
|---|---|---|---|---|---|---|---|
AP-Q5_K_XL | 76.75 | 35.76 | 112.51 | 5.46 | Q5_K 5.5 | 0.0839 ± 0.0008 | 86.63 ± 0.14 % |
AP-Q5_K_M (updated 2026-10-13) | 76.75 | 26.82 | 103.57 | 5.03 | Q5_K 5.5 | 0.0771 ± 0.0007 | 86.95 ± 0.14 % |
AP-Q4_K_XL (updated 2026-10-12) | 67.38 | 26.82 | 94.20 | 4.57 | Q4_K 4.5 | 0.0886 ± 0.0008 | 86.14 ± 0.14 % |
AP-Q4_K_M | 61.22 | 26.82 | 88.04 | 4.27 | IQ4_XS 4.25 + IQ3_S 3.44 | 0.1243 ± 0.0011 | 84.22 ± 0.15 % |
AP-IQ4_XS | 57.42 | 26.82 | 84.24 | 4.09 | IQ3_S 3.44 | 0.1584 ± 0.0014 | 82.89 ± 0.15 % |
AP-IQ3_XXS | 53.90 | 26.82 | 80.72 | 3.92 | IQ3_XXS 3.06 | 0.1844 ± 0.0016 | 81.85 ± 0.16 % |
AP-IQ2_S | 49.21 | 26.82 | 76.03 | 3.69 | IQ2_S 2.5 | 0.2657 ± 0.0022 | 79.14 ± 0.16 % |
Sizes in GiB. VRAM is what the GPU must hold; the 51.2B-parameter n-gram table
(per_layer_token_embd) is offloadable to system RAM with
-ot "per_layer_token_embd=CPU", so plan around the VRAM column, not the download.
Tier names follow the base ftype convention (as unsloth's do) and do not describe the
experts — the experts column does.
Against unsloth at the same quant level
Same reference, same corpora, same llama.cpp build for every row.
| file | VRAM | download | KL div held-out | top-1 held-out | KL div code | top-1 code |
|---|---|---|---|---|---|---|
AP-Q5_K_M | 76.75 | 103.57 | 0.0771 | 86.95 % | 0.0181 | 97.69 % |
AP-Q4_K_XL | 67.38 | 94.20 | 0.0886 | 86.14 % | 0.0206 | 97.38 % |
unsloth UD-Q4_K_XL | 76.86 | 103.69 | 0.0778 | 87.21 % | 0.0192 | 97.36 % |
At the same VRAM, AP-Q5_K_M matches unsloth's UD-Q4_K_XL on held-out KL divergence and is closer to the source on code;
unsloth keeps a slightly higher top-1 on held-out text. AP-Q4_K_XL needs 9.5 GiB less VRAM.
Which one should I use?
AP-Q5_K_M— the most accurate file here, if you have 77 GiB of VRAM.AP-Q4_K_XL— the default if 77 GiB doesn't fit: 9.4 GiB less VRAM for a KL divergence of 0.0886 instead of 0.0771.AP-Q4_K_M— 6 GiB less VRAM for 1.9 points of top-1. Take it if 67 GiB doesn't fit.AP-IQ2_S— the floor. Usable at 49 GiB, and 2.8 points ahead of the nearest 2-bit alternative at comparable size, but the gap to Q4 is large. Take it only if you must.
AP-Q5_K_M and AP-Q5_K_XL need identical VRAM (76.75 GiB). The updated AP-Q5_K_M is the more accurate
of the two; the XL's extra 9 GiB is n-gram-table precision only. Prefer AP-Q5_K_M.
Quality methodology
All figures are KL divergence and top-1 agreement against the same source, measured
on a held-out corpus the model has not memorised, -c 2048, via
llama-perplexity --kl-divergence. This means that numbers can't be directly compared with other measurements.
The code column uses a held-out code corpus the same way.
Why not wikitext. This architecture carries a 51.2B-parameter n-gram lookup table, and
wikitext is inside its memorised set. The same file scores 6.9–8.6 points higher top-1 on
wikitext than on held-out text, and the inflation grows as the quant gets smaller — so
wikitext compresses exactly the differences you are choosing between. It is vendor-
independent; other publishers' quants of this model show it too. For reference, AP-Q4_K_M
scores 84.22 % held-out and 91.82 % on wikitext.
Behaviour. AP-Q4_K_XL ran a loop stress test: 45 generations on prompts that provoke runaway reasoning (thinking on, --min-p 0.05, 12k-token cap). 0 loops; 44 of 45 finished within the cap.
Running
llama-cli -hf agentionai/Qwen3.8-Flash-Next-AP-GGUF:AP-Q4_K_XL \
-ngl 999 -fa 1 -ot "per_layer_token_embd=CPU"
Qwen's recommended settings:
| parameter | thinking mode | instruct (non-thinking) |
|---|---|---|
| temperature | 1.0 | 0.7 |
| top_p | 0.95 | 0.80 |
| top_k | 20 | 20 |
| min_p | 0.0 | 0.0 |
| presence_penalty | 0.0 | 1.5 |
| repetition_penalty | 1.0 | 1.0 |
Our behaviour tests run with --min-p 0.05, which we recommend for the smaller tiers.
The -ot flag keeps the 51.2B-parameter n-gram table in system RAM.
Vision
The vision projector is mmproj-F16.gguf at the repository root. Download it alongside any
tier to enable image input:
llama-cli -hf agentionai/Qwen3.8-Flash-Next-AP-GGUF:AP-Q4_K_XL \
--mmproj-hf agentionai/Qwen3.8-Flash-Next-AP-GGUF:mmproj-F16.gguf \
-ngl 999 -fa 1 -ot "per_layer_token_embd=CPU"
Changelog
- 2026-10-13:
AP-Q5_K_Mupdated. Same file name, size, VRAM and tensor types; better weights from an improved encode. KL divergence 0.0771 (held-out), 0.0181 (code). Download the file again; the first release stays available as revision86a35a81. - 2026-10-12:
AP-Q4_K_XLupdated. Same file name, size, VRAM and speed; better weights from an improved encode. KL divergence 0.0886 (held-out), 0.0206 (code). 0 loops in a 45-run loop stress test. Download the file again; the first release stays available as revision0061a60e.
Support AgentionAI
These quants are released freely. If it saves you compute or makes Qwen more useful, you can sponsor continued tuning, quantization and benchmarking on GitHub.