Qwen3.8-Whittle-16B
☕ Support this work
Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.
A 27B whittled down to 16.8B with a logit lens and a pricing table, then healed with one A100 evening. It now outscores every intermediate cut, including ones 4B larger, at 20 tokens/second on two consumer 8GB GPUs.
Whittle is Qwen3.8-27B minus 20 of its 64 layers (whole interval-blocks, chosen by measured boundary cost) and minus 25% of every remaining MLP's width (weakest neurons by ‖down_column‖ × activation-std). That is a −10.1B parameter cut performed with zero training, followed by a single 11M-token QLoRA heal on a fully clean-lineage mix (synthetic fact frames, programmatic short-think arithmetic, hand-written code drills, public-domain long pages).
Status
Earlier line (Aug 2026); the current Whittle models are the Whittle-Next / Whittle-Qwen-3.8 line: Whittle-Qwen-3.8-35B-A3B.
This model is a research preview that needs post training. It is published as the record of a compression method and its measurements rather than as a finished assistant: a compression-research artifact, evaluated with field measurements rather than academic benchmarks. Expect rough edges, use the serving settings below, and do not rely on it for factual reference or production systems. Further healing and instruction tuning runs are planned over time, and these checkpoints will improve as those land.
Non-starter for agent and long multi-turn use: repetition looping. In extended generations, multi-turn conversations, and agent loops (coding assistants, tool use) the model can fall into repetition loops badly enough to be unusable, even with DRY sampling enabled. Single-turn use with the recommended serving flags works well. Further training rounds targeting looping, multi-turn, and agent formats are actively in progress; this card will be updated as fixes land and pass real-world testing.
Which weights are which
Use the v2 weights: the -v2- GGUFs (gguf/Qwen3.8-Whittle-16B-v2-Q4_K_M.gguf and
gguf/Qwen3.8-Whittle-16B-v2-q8_0.gguf). v2 fixed code fencing (v1 emitted code outside markdown
blocks), reduced single-turn long-form loops and added three.js fluency; the numbers are under
Measured below. Serve with: --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 --repeat-penalty 1.15 --repeat-last-n 512. The v1 files remain for reproducibility.
| path | version | what it is |
|---|---|---|
gguf/Qwen3.8-Whittle-16B-v2-Q4_K_M.gguf, gguf/Qwen3.8-Whittle-16B-v2-q8_0.gguf | v2 (use these) | the v2 GGUFs for stock llama.cpp |
Qwen3.8-Whittle-16B-Q4_K_M.gguf, Qwen3.8-Whittle-16B-q8_0.gguf (repo root) | v1 | the v1 GGUFs, kept for reproducibility |
model-*.safetensors, config.json, tokenizer (repo root) | not stated | the healed weights in bf16 |
adapters/v1-heal/ | v1 | the heal adapter; applies to the un-repaired base |
adapters/v2-fencing-threejs/ | v2 | the fencing + three.js adapter; applies on top of v1 |
training/ | v2 | the v2 recipe and mix (heal_mix_v2.jsonl, heal_mix_v2_builder.py, train_heal_q38.py) |
research/ | v2 | the v2 probe results (q38_battery_v2.json, q38_battery2_v2.json, probe_v2.json, fence_loop_v2.json) and the fencing/loop test script (q38_fence_loop_test.py) |
docs/RESEARCH_activation_space_layer_merging.md | — | the research log |
Both heal adapters live in this repo under adapters/ (v1-heal applies to the un-repaired base;
v2-fencing-threejs applies on top of v1).
Measured
Battery and speed, the healed v1 against the cuts it came from (39 greedy tasks; RTX 4060 + 3050):
| params | file | battery (39 greedy tasks) | speed (RTX 4060 + 3050) | |
|---|---|---|---|---|
| Qwen3.8-27B (base) | 26.9B | n/a | not measurable on ref. hardware | n/a |
| 48-layer cut | 20.8B | 12.9GB | 33/39 | 5 t/s |
| un-repaired Whittle | 16.8B | 10.1GB | 25/39 | 20.5 t/s |
| Whittle (healed, this) | 16.8B | 10.1GB | 36/39 | 18.5–20.9 t/s |
The heal fixed 11 battery items and broke zero. Long-tail recall recovered
(recognition probe 2/7 → 5/7), arithmetic precedence and code completion returned to
textbook form, and the boiling point of water, the fragile fact that failed every
intermediate variant, answers correctly. Full measurement history, every pricing
run, and all scripts: see the companion research repo
Qwen3.8-p44w75-16.8B-unrepaired
and its research/ folder.
v2 against v1 (the v2 regression check; results in research/):
| check | v2 |
|---|---|
| code fencing suite | 8/8 (v1 emitted code outside markdown blocks) |
| single-turn long-form loops, mitigations disabled | 1/6 |
| battery (39 greedy tasks), regression held | 35/39 |
| recognition probe, regression held | 5/7 |
How it was made (short version)
- Sound the model: stream per-layer FP8 shards through an 8GB GPU, record each layer's identity cosine and a logit-lens sounding at every boundary.
- Price the cuts: block-drops must keep the GDN:attention interval (GGUF expressibility); price every candidate at the boundary, then at task level. Findings: drop damage is non-additive, width damage compounds, and layers 32–35 hold arithmetic, not knowledge.
- Cut: 20 layers + 25% of MLP width, zero training, single-width so stock tooling serves it.
- Heal: QLoRA r=64 on every linear (GDN projections included), 110 steps, cosine annealed to completion, clean-lineage data only.
- v2 heal: the code-fencing fix plus three.js fluency (trained on the official MIT examples + manual)
at the cost of one A100 hour on a 2.2M-token clean mix; recipe and mix in
training/, adapter inadapters/v2-fencing-threejs/.
Full methodology in the research log: docs/RESEARCH_activation_space_layer_merging.md in this repo
(the un-repaired repo carries its copy as research/RESEARCH_activation_space_layer_merging.md).
Run it
The GGUFs run on stock llama.cpp (any build with Qwen3.5-series support). Recommended serving command. The anti-loop sampling flags are part of the recipe, not optional garnish:
llama-server -m Qwen3.8-Whittle-16B-v2-Q4_K_M.gguf -ngl 99 -c 8192 --jinja \
--dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 4 \
--repeat-penalty 1.15 --repeat-last-n 512 \
--temp 0.7 --top-p 0.95 --min-p 0.05
In this repo that file is gguf/Qwen3.8-Whittle-16B-v2-Q4_K_M.gguf.
The Q4_K_M fits entirely in 16GB of VRAM (or 2×8GB split). This is a thinking model:
it reasons in <think> before answering, so give it generous max_tokens (500+),
and keep --jinja (included above) if your client uses tool calling. Context can be
raised well past 8192 cheaply: only 11 of the 44 layers are full attention, the
rest carry fixed-size recurrent state.
Caveats
- Open-ended creative prompts (e.g. "write a haiku") can exhaust the thinking budget in deliberation. Concrete instructions ("exactly 3 short lines, no preamble") work. This is the one measured failure the heal did not close; it is targeted in the next round.
- Evaluated with a 39-prompt greedy battery plus probes, not academic benchmarks. The numbers above are honest field measurements, comparable within this table.
- The heal mix is small and targeted. Knowledge breadth beyond what a 16.8B carries is not magically restored: this is an efficient model, not a 27B in disguise.
- Repetition looping in extended, multi-turn and agent use: see Status above.
Honest state of the model, and what it would take to finish it
Web development capability took real damage in the compression: current measurements show the shape cut collapsed targeted web-token retrieval roughly 28-fold at the boundary, and we are still isolating how much came from the depth cut versus the width prune (restoration pricing runs are in progress). The heals so far were single A100 hours and evenings; they recovered facts, arithmetic, and code fencing, but turning this research preview into an actually usable daily model (agent-capable, loop-free in long multi-turn use, restored web-dev depth) needs sustained post-training that is beyond a grocery-money compute budget. The methods and data are ready; the A100 hours are the missing ingredient.
Support this work
Independent research on consumer hardware. Every donation becomes A100 hours, and every A100 hour ends up as a public model or a public measurement. If you want the usable version of this model to exist, this is the lever. ☕ ko-fi.com/davida81328
Provenance, licence and authors
Base model by the Qwen team (Apache 2.0). Whittled and healed by David Aylward with Claude (Fable 5, Anthropic) as co-author. The instruments, pricing runs, builds, training and evaluations were executed by Claude under David's direction, including several load-bearing ideas of David's: the parallel-composition merge operator, the slice-stack-merge width reduction, and the recognition-vs-recall damage probe.