Whittle-Qwen-3.8-45B-A3B — GGUF

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

The current Whittle flagship, Whittle-Qwen-3.8-35B-A3B, with the 76 experts per layer that its expert prune removed put back from Qwen3.6-35B-A3B. The body is back to 256 routed experts per layer, the 10 B n-gram memory is kept, and nothing was retrained after the splice. 45.1 B total (35.1 B body + 10.0 B memory, counted from this GGUF), ~3 B active (8 of 256 routed experts + the shared expert), Qwen3.8-Flash-Next (qwen4_exp) format, runs on stock llama.cpp.

Against the 35B it knows more long-tail facts, codes at least as well and scores 96/96 on our tool-use probes; it is weaker on arithmetic word problems and needs 5.8 GB more memory. Details under Measured.

Status

Experimental (4 Oct 2026). An untrained splice of two published models, measured on the same battery as the 35B and its base. The restored experts were trained for Qwen3.6's own body, not for this one, and have not been healed into it; the GSM-50 drop below is the visible cost. A short re-heal is the next step and needs paid GPU time (see the support note above). For the safest choice today, use the 35B.

Which file

The memory table is about 5.6 GB of the Q4_K_M (10.5 GB of the Q8_0). It is a lookup, one row per token per head, so keep it in system RAM with -ot per_layer_token_embd=CPU: the GPU then holds the 256-expert body (21.5 GB at Q4_K_M). The K-quants were requantised from the Q8_0. No Q6_K yet.

Measured (4 Oct 2026, on the Q4_K_M)

Stock llama.cpp on two T4 GPUs, one sample per item. The knowledge rows are cloze prompts scored on a greedy raw completion; everything else goes through the chat endpoint with the card sampler (temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05) and thinking on. The other columns ran on the same harness: the 35B at Q6_K (its EvalPlus at Q5_K_M) and the base Qwen3.6-35B-A3B at Q4_K_M (bartowski, importance matrix).

What is real and what is noise. Paired on the same items with an exact test, against the 35B: the long-tail names gain is real (18 gained, 1 lost; p < 0.001) and so is the GSM-50 drop (6 lost, none gained; p = 0.031). Coding is level: 24 tasks gained and 21 lost on HumanEval, 34 against 19 on MBPP+ (p = 0.053). The web-page replay drop (34 to 27 of 40) is within single-run noise (p = 0.11), but the 2 invented tool calls are new: neither the 35B nor the base made any. The 35B and base columns ran at different quantisations, and the base with Whittle's sampler rather than Qwen's own 0.6 / 0.95.

The chart compares the four Whittle builds that share one body: the 25B (no table), the 35B (+ the table), the x256 (+ the restored experts, no table) and this 45B (both). The table buys code and long-context tool use; the restored experts buy long-tail knowledge; only the two together lost GSM-50. The full comparison, with all five models, is on the 35B card.

How it was made

1. Start: the published Q8_0 of Whittle-Qwen-3.8-35B-A3B, Phase-2 step 32010 (180 routed experts per layer + the 10 B memory).
2. Experts: the 2 Sep 2026 prune removed 76 of Qwen3.6-35B-A3B's 256 routed experts in every layer (the keep lists are in eval/prune_report.json). Those 76 were read from Qwen3.6's bf16 weights, quantised to Q8_0 and appended to each layer as experts 180–255.
3. Router: their original router rows were appended too, scaled per layer by the median norm ratio between the trained and the original rows of the kept experts (1.001–1.026). Five weeks of training had barely moved the router: trained rows sit at cosine 0.98–0.995 to the originals.
4. No training. The file was written by streaming byte surgery (header patched in place, every other tensor copied), checked by comparing a kept expert with its original (gate cosine 0.990) and smoke-tested on stock llama.cpp, then requantised.

Run it

llama-server -m Whittle-Qwen-3.8-45B-A3B-Q4_K_M.gguf -ngl 99 -c 16384 --jinja -fa on -ot per_layer_token_embd=CPU

• Sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05: sample, do not decode greedily; greedy decoding loops on this family.
• Thinking: "chat_template_kwargs": {"enable_thinking": true}. Give it max_tokens 4096+ for code and 8192+ for maths (9 of 60 MATH problems hit 6,144). --reasoning-format deepseek separates the thinking into reasoning_content.
• On llama.cpp builds that have --lazy-mode (newer than ~30 Aug 2026), add --lazy-mode off so the memory table is loaded into RAM; otherwise it is read lazily from disk and -ot is ignored, which makes prompt processing slow.
• To move the routed experts to RAM as well on small cards: -ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU".
• Tool use / agents: keep each assistant turn's reasoning_content in the history for the rest of the tool episode. The chat template is the 35B's, which accepts a system message anywhere, so Claude Code works against llama-server's /v1/messages.
• Architecture qwen4exp; if your build reports an unknown architecture, update llama.cpp.

Caveats, measured

• Arithmetic word problems are weaker than the 35B (GSM-50 38 against 44; 8 wrong answers and 4 that ran out of tokens). If that is your workload, use the 35B.
• llama.cpp's web page: 2 invented tool calls in 40 replays of a real chat. Agents with real tools were fine (96/96 on our probes).
• Exact-answer cloze is lower (PopQA exact match 52/288 against 71/288) even though name completion is higher (111/252 against 94/252); the two columns were measured at different quantisations (Q4_K_M against Q6_K).
• Every number is one run, one sample per item: a few items either way on GSM-50, MATH-60 or the chats is noise.
• The base Qwen3.6-35B-A3B still knows more long-tail names (132/252).
• Serve the table whole: the body depends on the memory (see the 35B card); a build that drops or re-hashes per_layer_token_embd loses code and long-context tool use.

Provenance

David Aylward (logic65) & Claude (Anthropic). Built from logic65/Whittle-Qwen-3.8-35B-A3B (Phase-2 step 32010; its card carries the full lineage, teacher Qwen/Qwen3.8-27B, memory contents from Qwen/Qwen3.8-Flash-Next) and the routed experts of Qwen/Qwen3.6-35B-A3B. All Apache-2.0. Results files: the battery and EvalPlus runs of 2–4 Oct 2026 (Kaggle, two T4 GPUs).