Qwen3.8-p44w75 — a 16.8B cut of Qwen3.8-27B (zero training, ⚠️ UN-REPAIRED)

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

27B → 16.8B → 20.5 tok/s on two consumer 8GB GPUs. No fine-tuning, no distillation, no calibration training — pure measured surgery, built in one day on a Ryzen 2700X.

⚠️ This is the un-repaired research artifact. It has known, documented damage (see Caveats). It chats, codes, and does arithmetic correctly, but some long-tail factual associations were corrupted by pruning and it can ruminate on open-ended prompts. The healed version that was planned has since been released (see Status). We release the raw cut because the damage profile is itself the interesting scientific object.

Co-authored by David Aylward and Claude (Fable 5, Anthropic), who ran the measurement campaign, surgery, and evaluations described below.

Status

Earlier line (Aug 2026); the current Whittle models are the Whittle-Next / Whittle-Qwen-3.8 line: Whittle-Qwen-3.8-35B-A3B.

🪚 The healed release of this cut is Qwen3.8-Whittle-16B — same 16.8B, one QLoRA evening later: 36/39 on the battery (this un-repaired base scores 25/39), water boils at 100°C again, fibonacci is textbook. This repo remains the research artifact: the raw cut, the full measurement history, and every script.

This checkpoint needs post training. It is a research preview, published as the record of a compression method and its measurements rather than as a finished assistant. Further healing and instruction tuning runs are planned over time, and these checkpoints will improve as those land.

What this is

Qwen3.8-27B is a 64-layer Gated DeltaNet/full-attention hybrid (3:1 interval pattern). This model is what remains after two orthogonal, individually-measured cuts:

1. Depth: 20 layers deleted in whole interval-blocks of 4 (layers 4–11, 24–31, 32–35) — keeping the stack exactly 3:1 so it stays expressible in GGUF with zero inference-code changes. Block choice was priced empirically with a streaming logit-lens over per-layer FP8 shards on an 8GB GPU.
2. Width: every remaining MLP sliced from 17408 → 13056 neurons (uniform 25% prune, all 44 layers including layer 0 and the tail), keeping the strongest 75% by ‖down_column‖ × activation_std measured over real corpus tokens.

Result: 44 layers, FFN 13056, hidden 5120, vocab 248k, untied head — 16.8B parameters (from 26.9B).

Which weights are which

Also in this repo: research/ (the scripts and measurement JSONs, see Reproduction), training/ (the heal recipe: train_heal_q38.py, heal_mix_builder.py, heal_mix_q38_clean.jsonl, colab_heal_cell.txt) and docs/ plus research/RESEARCH_activation_space_layer_merging.md (the research log).

Measured (39-prompt greedy battery + chat)

Chat mode: answers factual and computational questions correctly ("What is 17*23?" → thinks "340 + 51 = 391" → answers "391"); very open-ended prompts can exhaust the thinking budget.

How it was made (compressed — full research log in this repo's research/)

• Instrument first: a streaming prober loads one FP8 layer shard at a time onto an 8GB GPU, records each layer's identity cosine and a logit-lens sounding at every boundary. Note this model's head is untied — lens through lm_head, not the embedding.
• Operators measured on identical layer groups: weight-space merging (LaCo-style) collapses on this GDN hybrid (stream cos 0.27); activation-space parallel composition holds 0.93; plain drop 0.75. Weight-space neuron twins are absent (max cosine 0.58); functional twins are pairwise-abundant. Redundancy in this trained hybrid is functional and pairwise, not literal or blocky.
• Depth ladder: block-drop damage is non-additive — deleting both cheap 8-layer bands costs no more than either alone. 48L and 44L variants both survive at task level; 40L breaks structurally. The tested wall for zero-training depth removal sits between −7.7B and −9.2B.
• Width ladder: 25% width prune rides nearly free end-to-end; 50% breaks. Not because per-layer cost is high (halving one MLP costs 0.5–1.5% of the stream) but because ~35 boundaries compound.
• Expert-sliceability (negative result): activation mass is uniformly diffuse across any balanced clustering at every depth tested — top-2-of-8 oracle routing captures 27% of mass where chance is 25%. Zero-training MoEfication has no raw material here; sparsity would have to be trained in.
• Evaluation at every rung: greedy task batteries (facts / code / arithmetic / commonsense) + chat probes, served through llama.cpp — not just stream cosine. Stream metrics predicted where damage lands but only task testing revealed which capability pays (e.g. layers 32–35 turned out to hold arithmetic/code precision, not knowledge).

Reproduction

The research/ folder contains the scripts used: the streaming lens prober, per-layer stats collection, masked pricing passes, the surgery builders, and the task batteries — everything runs on a single 8GB GPU against the per-layer FP8 shards of the base model.

Run it

The GGUFs run on stock llama.cpp — any build with Qwen3.5-series support (mainline since models: support qwen3.5 series). Suggested serving on 2×8GB:

llama-server -m q38p44w75_q4km.gguf -ngl 99 -c 4096 --port 8991

Notes: recent llama.cpp splits llama-cli (conversation-only) from llama-completion (raw completions). Requantizing from the q8 GGUF needs --allow-requantize.

Converting the HF checkpoint yourself: two caveats. (1) convert_hf_to_gguf.py may not recognize Qwen3_5ForCausalLM — register it as an alias of the Qwen3.5 text model class. (2) transformers fabricates a default text_config (with the 2B sibling's geometry) over flat text-only qwen3_5 configs, and the converter merges text_config over the root — this repo's config.json ships with an explicit mirrored text_config to defuse that; keep it if you edit the config.

Caveats (read before using)

• Corrupted long-tail facts. Width pruning removes rarely-firing neurons, which is where long-tail knowledge lives. Recognition probes show ~⅔ of the lost facts are rewired to plausible-wrong neighbors, not merely attenuated: this model will confidently tell you Buzz Aldrin was first on the moon, water boils at 90°C, and the solar system has nine planets. Do not use for factual reference.
• Think-budget rumination on open-ended/creative prompts (asked for a haiku, it may deliberate about haiku rules until the token limit). Concrete instructions work better.
• Evaluated with a 39-prompt battery and short greedy completions — no MMLU/GSM8K yet; treat all numbers as field measurements, not benchmarks.
• Single model, single run of everything; the intact 27B was never run on the same battery (it does not fit the reference hardware), so the true ceiling is unknown.

Support this work

Independent research on consumer hardware — every donation becomes A100 hours, and every A100 hour ends up as a public model or a public measurement. ☕ ko-fi.com/davida81328

Provenance, licence and authors

Base model by the Qwen team (Apache 2.0). Surgery, measurement campaign, and this card by David Aylward with Claude (Fable 5, Anthropic) as co-author — the measurement instruments, pricing runs, builds, and evaluations were executed by Claude under David's direction, including several of the load-bearing ideas (parallel composition, slice-stack-merge, the recognition-vs-recall damage probe).