Qwen3.8-Flash-Next 125B-A5B INT4-AR
Qwen3.8-Flash-Next with 5 routed experts per token instead of 10, healed so it stays close to the original, quantized to int4. It runs on one DGX Spark (GB10, 128 GB) at roughly 70-76 tokens/s.
125B parameters in total, 4.8B active per token. The original activates 6B.
Everything needed to serve it is in this one repository, including the 49 GB FP8 n-gram table under
ple-table/. Nothing else to download.
Run it
git clone https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST.git && cd Qwen3.8-Flash-Next-Int4-FAST && bash setup.sh
That builds the serving image, downloads this repository, and starts an OpenAI-compatible server on port 8000. The scripts and the full explanation are in that repo.
Serving by hand needs Saren-Arterius/qwen3.8-Flash-DGX-AutoRound,
because a stock vLLM cannot serve this checkpoint's int4 + fp8 mix. Point VLLM_PLE_MMAP_DIR at
ple-table/.
How it compares
Measured on one DGX Spark with our own harness, so read it as a comparison rather than a leaderboard entry. Capability is a trimmed average over 6 runs; tool use is 3 trials.
| This model | Original, 10 experts | |
|---|---|---|
| Capability | 49.3 | 51.8 |
| Tool use | 91 | 91-92 |
| Speed | ~70-78 tok/s | ~57 tok/s |
| Active parameters | 4.8B | 6.0B |
Capability was measured over 6 runs with an int8 head at 51.8, and the int4 head shipped here at 49.3; the two are within noise of each other, so read it as 50% either way. The int4 head is worth about 8% more speed.
Tool use survives the cut intact. General capability gives up about four points. In exchange you get a fifth fewer active parameters and noticeably faster generation.
What was done to it
The routed experts were cut from top-10 to top-5, which costs quality because each token now sees half the expert capacity it was trained with. That loss was then healed by training only the shared expert (37.75M parameters, everything else frozen) to match what the uncut model would have produced: pure KL self-distillation, the same model at top-10 acting as its own teacher, over 14,667 examples of code, maths, general text, tool calls and long context.
Healing recovered most of the cut. Unhealed, the capability score sat at 41.8; healed, it reaches 49.3 against the original's 51.8. Text supervised fine-tuning was tried first and plateaued at 42.8, so the distillation objective is what made the difference.
What is inside
| Routed experts, 48 layers x 512 | int4 GPTQ-Marlin, group 128 |
| lm_head | int4 GPTQ-Marlin, group 128, tuned with AutoRound |
| Attention and gated-delta projections, shared expert | blockwise fp8 e4m3 |
| Embeddings, routers, norms, hyper-connections | bf16 |
ple-table/ | the 51B-parameter n-gram table, fp8, read from disk by memory map |
The lm_head was requantized from the original bf16 weights with AutoRound (1,000 iterations, calibration captured through this healed model). Against the bf16 head it keeps 97.2% top-1 agreement; the int8 head it replaces kept 99.6%, but it is half the size and the head is read four times per decoding step, so the trade buys real speed.
Also here: medium_chat_template.jinja and xhigh_chat_template.jinja, two optional chat templates.
The default chat_template.jinja is the stock Qwen one. The medium variant stops injecting the
"reasoning effort is xhigh" instruction, which cut benchmark time and tokens by about a quarter at the
same score; both variants add three tool-use rules and accept <|think_off|> in the prompt.
Credits
- Qwen for Qwen3.8-Flash-Next
- Intel for the AutoRound int4 quantization
- Saren-Arterius for the hybrid checkpoint and the vLLM build that serves it on a Spark, and for the fp8 n-gram table included here
The expert cut and the healing are ours. Everything else is theirs.
README_upstream_hybrid.md and ple-table/README_upstream_table.md are the original model cards of the
two checkpoints this was built from, kept as they were.