Qwen3.8-2.4T-A95B, expert-pruned to fit in 512 GB of memory.
The unpruned Q2 quant of Qwen3.8 (UD-IQ2_XXS, 656.6 GB) cannot run on a 512 GiB machine at all — it exceeds total RAM. This build prunes the experts an English + code deployment doesn't route to, keeping the healthier Q2 quantization intact: 404 GB (376 GiB), fully resident on a 512 GiB Mac at ~9.5 tok/s.
No fork, no PR branch, no custom runtime. This file loads and generates
on stock mainline llama.cpp — verified by running it there (greedy outputs
identical to our fork). Download and llama-server, nothing else.
| experts | 304 of 512 per MoE layer (uniform), selected by measured routing counts; width untouched (unlike our 256 GB sibling) |
| calibration | English web + code corpus (200k tokens) through the unpruned model; per-(layer, expert) router selection counts |
| selection coverage | 89.7% of routing selections retained (worst layer 83.2%) |
| quantization | untouched — byte-slab copy of UD-IQ2_XXS (layer-mixed ~1.9 bpw experts), no requantization |
| size | 404 GB = 376 GiB — full -ngl 99 offload fits under macOS's Metal working-set ceiling with room for KV |
| KLD vs unpruned UD-IQ2_XXS | held-out en: mean 0.110, median 0.024, argmax agreement 87.6%, PPL 8.70 → 9.31 (×1.07) · held-out code: mean 0.196, median 0.007, argmax agreement 90.0%, PPL 1.75 → 2.03 (×1.16) |
Expert selection is count-based (router hit counts), not gate-weighted REAP saliency — on Kimi-K3 this shortcut recovered 90.9% of saliency mass vs 93.5%. The counts come from the same measured workload as the 256 GB sibling; the two builds share their keep-set.
Like every calibration-pruned build: what the corpus leaves out is what gets deleted. Non-English languages and off-domain abilities are deliberately sacrificed.
Provenance: the exact pruning plan is published — plans/reap_plan_qwen_304.json (sha256 299b756b01231c43…, full hash in SHA256SUMS); plan + source quant + the MIT slicer reproduce this file's bytes. Split-half recalibration (same-distribution halves): keep-set overlap 92.7%, count-mass retention 98.7–98.9% (worst layer 97.0%) — details in plans/README.md.
Reading the quality numbers
Measured exactly as on the 256 GB sibling's card (teacher-forced held-out text, ~128k tokens per domain, base logits from the unpruned quant): the numbers answer how far does this build drift from the model it was cut from. Median KLD 0.007 on code means half of all code tokens are essentially untouched; the top-1 token survives 90% of the time. Side-by-side with the 256 GB build (which also cuts expert width and starts from a 1.56 bpw base), this build keeps roughly twice the fidelity on every metric — en argmax 87.6% vs 79.2%, code 90.0% vs 86.6%, PPL overhead ×1.07-1.16 vs ×1.20-1.26. That gap is the measured price of the last 158 GB.
Verified vs. not verified
Verified:
| claim | evidence |
|---|---|
| KLD / argmax agreement vs the unpruned quant | table above, held-out en and code |
| Slicing is lossless for surviving weights | identity prune byte-identical — pinned by tests in the tooling repo |
| Loads and generates | stock mainline llama.cpp (9.3 tok/s) and our fork (9.5 tok/s), greedy outputs identical, full -ngl 99 resident on an M3 Ultra 512 GB |
| Agentic sanity check | two labeled conditions, per our protocol: bare prompt 1/3 — two one-turn deaths obeying the benchmark's phantom ```python scaffold instruction (log-verified, same trap class as the parent's single bench fail) · with the standard counter-note (promptv1m) 3/3 — the trapped cells become normal grind-passes (20-min death → 74-min $1,000 win). Full data: swelancer-local-subset-evals |
What the sanity check actually demonstrates — and what most quant/prune releases never show: this build drove Qwen's own CLI agent through three real, paid SWE-Lancer tasks end-to-end — multi-hour tool-calling sessions (29 / 74 / 185 minutes) against a real React Native codebase — and solved all three ($2,000 in original prize value) under the disclosed conditions. Not a perplexity table: the model plans, edits files, runs shells, and finishes.
An owner-requested extension then ran the five hard tasks (the ones a 2-bit K2.7 baseline failed): 2/5, $6,000 — including the heaviest task in the set (a ~104k-token issue) solved in 99 minutes where the streamed unpruned parent needed 183, resident speed converting directly into solve time. Total across all eight: 5/8, $8,000 vs the parent's 7/8 — the KLD ladder (parent > this build > the 256 GB sibling) reproduces exactly as a bench ladder (7/8 > 5/8 > 3/8). Fidelity costs capability; this build is the measured middle point, and every cell is public: swelancer-local-subset-evals.
Verified operating envelope — every number on this card was measured at a stated point: quality (KLD / argmax / PPL) on 2048-token teacher-forced windows — the same regime the expert selection was calibrated in; agentic work served at context 131,072 through multi-hour sessions (up to 185 min). The model's trained window is 262,144 (GGUF metadata). Beyond these points — longer contexts, non-English work (deliberately cut by calibration), off-corpus domains — you are past our measurements.
Run
llama-server -m Qwen3.8-2.4T-A95B-REAP-512GB-IQ2_XXS.gguf \
--port 8090 -ngl 99 -c 131072 --jinja \
--temp 1.0 --top-p 0.95 --top-k 20
Works on stock mainline llama.cpp (no fork needed; Qwen3.5-MoE support). Sampling per the Unsloth card; hybrid thinking model.
How it was made
Same pipeline as the 256 GB sibling (tooling, MIT), minus the width cut: count the router selections over the calibration corpus, keep the top 304 experts per layer, slice the GGUF as raw byte slabs, renumber the router. blk.92 (a stored, never-executed MTP block) is sliced blind to keep the file loadable. No requantization anywhere.
Credits: Qwen team, Unsloth (UD-IQ2_XXS this inherits), Cerebras REAP, kimi-k3-mlx.
日本語の説明
Qwen3.8のQ2量子化(UD-IQ2_XXS、656.6GB)は512GiB機のRAM総量を超えていて
そもそも載りません。これは英語+コード用途でほぼ呼ばれないexpertを実測
ルーティングで削って404GB(376GiB)にした版で、512GiB機に-ngl 99で
全載せ・約9.5 tok/sで動きます。幅は削っていないので、256GB版より忠実度が
全指標で約2倍良い(code argmax一致90.0%、KLD中央値0.007)。多言語は設計上
壊れています。素のmainline llama.cppで動作検証済み(fork不要)。単なる
perplexity表ではなく、Qwen公式CLIで実在の有償SWE-Lancerタスク3問を
数時間のツール往復の末に3問とも解き切ったことを検証済みです。