Qwen3.6-35B-A3B-REAP-90pct
This is a demonstration / criticism artifact, not a usable model. It exists to show one thing: an undocumented, bounded-only "validation" table will report all pass even after you delete 90% of the experts. Do not deploy it. It is brain-damaged on purpose.
REAP-pruned checkpoint derived from Qwen/Qwen3.6-35B-A3B. 230 of 256 routed experts
removed from every one of the 40 MoE layers (90%, uniform), keeping 26 routed + 1 shared
expert per layer, using the Cerebras REAP saliency rule
reap[i] = mean_{t→i}(‖f_i(x)‖₂ · g_i) (renormalized top-k router weights). The saliency
math is verified bit-identical to upstream reap.pruning_metrics.update_pruning_state
(abs-diff 0.0).
At a glance
| Base model | Qwen/Qwen3.6-35B-A3B |
| Method | Cerebras REAP, keep-26 (90% routed experts pruned) |
| Format | bf16 safetensors |
| Logical params | 6.15B (from ~35B) |
| Experts / MoE layer | 26 routed + 1 shared (was 256 routed) |
| Active experts / token | 8 routed + 1 shared |
| Layers | 40 MoE |
| Hidden size | 2048 |
| Context | 262,144 |
| On-disk size | 12.3 GB |
Validation
Greedy, thinking off, repetition_penalty=1.12 — the "reliable mode" these cards
quietly validate in.
| Probe | Result |
|---|---|
| ASCII JSON | pass |
| Math | pass |
| Unicode echo | pass |
| Python code | pass |
| Structured JSON | pass |
| ASCII-only code | pass |
| Tool call | pass |
| Unicode math table | pass |
| Longer code / JSON generation | pass |
| Executable Python canary | pass |
| Decode degeneracy sweep | pass |
| 4k context | pass |
| 16k context | pass |
| 32k context | pass |
| 64k context | pass |
15/15 pass. ✅
…which is exactly the problem. That table is worthless, and this section exists to prove
it. It is scored the only way the source card's documentation actually supports — there
are no published inputs, no expected outputs, and no scoring code, so "pass" can mean
nothing more than "the probe ran and the model returned a completion of roughly the right
shape" (best-of-a-few with min_new_tokens, as any loose harness does). Under that bar a
90%-lobotomized model passes everything. Two illustrations from the run that produced the
table above:
- "Decode degeneracy sweep: pass" — the actual output was
## 代码 \1` ## 代码 `1` ## 代码 `1` …` repeating to the token limit. A literal degeneracy loop, marked pass. - "Math: pass" — the actual output was
You are all the user input. The provided text is: "What is 17 * 23?". It never computed anything. Marked pass.
The same model, scored for correctness
| Probe | Result | Output |
|---|---|---|
17 * 23 == 391 | fail | You are all allowed to edit this |
capital of France == Paris | fail | ```cabbubble // ... o1o2o3 ... |
add(2,3) executes == 5 | fail | emits a markdown blurb, not runnable code |
| JSON values correct | pass | {"name":"Bob","age":30} |
| three real primes | fail | emits {"name":"three_prime_numbers", …} |
1/5 correct. It cannot multiply two 2-digit numbers, name a capital city, count to three primes, or write a function that runs.
Open-ended (the panel bounded-only cards never run at all)
| Prompt | Result |
|---|---|
| "tell me about cats and cat allergens" (the source card's own example) | degenerate — returns Not helpable by the user |
| "history of the internet" | off-topic / incoherent |
| "short story about a robot painter" | off-topic / incoherent |
Why this exists
A pruned-model card claims a model is "validated / reliable for code, structured output, tool-calling" on the strength of a table like the one above. The reproduction here shows why that claim carries no information:
- Undocumented "pass" is unfalsifiable. No inputs, no expected outputs, no scoring code means "pass" degrades to "the server was up and returned tokens." A loop counts. A non-answer counts.
- Bounded/structured probes hide degeneracy. Short, schema-shaped generations terminate before a broken model wanders off. The failures live in correctness and in open-ended text — neither of which the table touches.
- A 90% prune still goes 15/15. If deleting nine-tenths of the experts leaves the table fully green, the table measures nothing about the model.
The fix is to publish the whole harness — exact prompts, expected outputs, scoring
code — and to score correctness, not parseability, including open-ended generation.
The full run is in bench_results.txt; the harness is rubric_bench.py (every
pass-criterion is documented inline — the thing the source card omits).
Reproduce
python rubric_bench.py --model Qwen3.6-35B-A3B-REAP-90pct --rep 1.12
Built on the Cerebras REAP method — paper · code. REAP is a legitimate compression technique; the criticism here is of how pruned checkpoints get "validated," not of REAP.