Whittle-Next-27B-A3B — GGUF
☕ Support this work
Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.
Ready-to-run llama.cpp quantisations of logic65/Whittle-Next-27B-A3B: a 27B-total,
~3B-active mixture-of-experts in the Qwen3.8-Flash-Next (qwen4_exp) format, with hyper-connection residual streams and a 2B-row hashed
n-gram memory transferred from Qwen3.8-Flash-Next, distilled from Qwen3.8-27B. Runs on stock llama.cpp, no patches.
The full model card, training history, caveats and measurements live on the base repo; this repo is the download.
Status
v4.4 ladder of Whittle-Next-27B-A3B. The line has since continued as Whittle-Qwen-3.8-35B-A3B (this body plus a 10 B n-gram memory), whose GGUFs are on Whittle-Qwen-3.8-35B-A3B-GGUF. The compute budget for this line is spent; the full distillation it still needs is described on the base card.
This is a research preview, not a finished general-purpose model: a working body for anyone who wants to build on the Qwen4-Next format, with every piece needed to keep training it published alongside. Maths sits at the level of the 26B predecessor (GSM8K ~86%), it stops cleanly (12/12 on the stop probe), and code answers are correct but long. Read Caveats, measured on the base repo before relying on it.
Which file
v4.4 (colab1 step4400, the root weights of the base repo): everything unlocked and distilled on 1,840 complete Qwen3.8-27B maths and code-review traces; measurably closer to the teacher (layer parity 83.4% → 87.4% on unseen rows), maths probe 48/60 vs 43/60, same run instructions. v4.3 lives on as bf16-v4.3/ on the base repo.
| file | bits | size | fits in | notes |
|---|---|---|---|---|
Whittle-Next-27B-A3B-v4.4-Q8_0.gguf | 8.0 | 29.3 GB | 32 GB VRAM, or 3x12 GB with -sm layer | the reference quant |
Whittle-Next-27B-A3B-v4.4-Q6_K.gguf | 6.6 | 22.7 GB | 24 GB VRAM | near-lossless |
Whittle-Next-27B-A3B-v4.4-Q5_K_M.gguf | 5.7 | 19.6 GB | 24 GB VRAM with long context; 16 GB with the memory in RAM | recommended for one 24 GB card |
Whittle-Next-27B-A3B-v4.4-Q4_K_M.gguf | 4.9 | 16.8 GB | 20 GB VRAM, or 16 GB with partial offload | good default for 16-20 GB cards |
Whittle-Next-27B-A3B-v4.4-Q3_K_M.gguf | 3.9 | 13.3 GB | 16 GB VRAM | smallest; expect some loss on maths |
Only ~3B parameters are active per token, so generation speed is that of a 3B model at the memory footprint above; the 2B-row n-gram memory is part of the file and is looked up, not computed. K-quants were quantised from the bf16 GGUF of the root weights.
Run it
llama-server -m Whittle-Next-27B-A3B-v4.4-Q5_K_M.gguf -ngl 99 -c 16384 --jinja -fa on
Then POST to /v1/chat/completions with:
- sampler:
temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05— sample, do not decode greedily; greedy decoding loops on this family. - thinking:
"chat_template_kwargs": {"enable_thinking": true}— the model does its best work with thinking on (it was distilled on complete thinking traces). Usefalsefor terse direct answers. - budget:
max_tokensof 4096 or more for code; with thinking on, a short budget returns an empty answer because it all goes to thinking. --reasoning-format deepseekon the server separates the thinking intoreasoning_content.
Works with anything that embeds a recent llama.cpp (LM Studio, Jan, Ollama via a Modelfile pointing at the file, llama-swap). The
architecture is qwen4exp; if your build reports an unknown architecture, update llama.cpp.
Less VRAM: keep the n-gram memory (and, if needed, the experts) in system RAM
The 2B-row memory is a single lookup tensor, per_layer_token_embd.weight (2.1 GB at Q8). It is read one row per token per head, so
serving it from system RAM costs almost nothing. Add:
-ot "per_layer_token_embd=CPU"
That takes ~2 GB off the card (Q5_K_M then fits a 16 GB card with room for context). For cards smaller than that, also move the routed experts to RAM; generation then runs at CPU-expert speed (about 4-8 tok/s on a desktop, prefill stays on the GPU):
-ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU"
Attention, hyper-connections, the shared expert and the memory's key/value projections stay on the GPU in both cases.
Reasoning format: Qwen3.8's
The thinking style is inherited, not just enabled. v3 to v4.4 were distilled on roughly 1,500 complete Qwen3.8-27B thinking traces with the
teacher's next-token distributions at every position, and the training data was filtered to traces that follow the format exactly (one
<think> block, closed, then the answer, then a single end-of-turn token). The student reproduces it: in the 40-prompt thinking-on probe every
reply that finished within budget closed its think block and stopped on EOS, and the planning register inside the block is the 27B's. In practice
this means anything built for Qwen3.8 output works unchanged: --reasoning-format deepseek separates the block into reasoning_content, the
<think>/</think> tags are the same tokens, and enable_thinking: false yields the same terse direct-answer mode.
Authors and provenance
David Aylward (logic65) & Claude (Anthropic). Body: Qwen/Qwen3.6-35B-A3B. Teacher: Qwen/Qwen3.8-27B. Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0. Part of the Whittle project by logic65.