SCOUT — Qwen3-4B @ 70% unstructured sparsity (δ = 0.02)
Qwen3-4B pruned to 70% unstructured sparsity by SCOUT: sparsity is grown by projected-gradient mask updates that are gated by a measured self-KL trust region, while the weights recover under a combined next-token / offline-KD / on-policy-KD objective against the dense teacher.
⚠️ Which run is this?
Five Qwen3-4B 70% ablation arms were uploaded with near-identical repo names
differing only by timestamp. This repo is the headline arm — A0, the full
method at δ = 0.02, the best-scoring 70% cell:
| repo (timestamp suffix) | arm | avg5 |
|---|---|---|
…_20260901_080954 ← this repo | A0 — SCOUT, δ = 0.02 (full method) | 45.58 |
…_20260903_081638 | B1 — w/o rollout refresh (frozen dense pool) | 43.97 |
…_20260903_144159 | A2m — schedule k-rule, pace-matched | 42.75 |
…_20260903_072918 | A3 — w/o trust region (δ = ∞) | 41.87 |
…_20260903_072500 | A1 — SCOUT, δ = 0.01 | 41.20 |
All rows: Qwen3-4B, 70% unstructured, lr 1e-4, 2048 steps, everything except the named component held identical. Single seed (42), one run per cell.
Results
Evaluated with lighteval (quick profile), 5 tasks:
| Task | Metric | Score |
|---|---|---|
| MATH-500 | accuracy † | 72.40 |
| GSM8K | extractive_match | 71.49 |
| GPQA-Diamond | gpqa_pass@k:k=1 | 35.86 |
| IFEval | prompt_level_strict_acc | 39.93 |
| LiveCodeBench (codegen) | codegen_pass@1:16 | 8.21 |
| avg5 | mean of the above | 45.58 |
| WikiText-2 | perplexity | 16.85 |
† MATH-500 is quoted from the source ablation table; the other four scores were
re-read from this repo's own lighteval result JSONs and match that table exactly.
Per-task raw lighteval outputs (results/*.json + details/*.parquet) are
included in this repo under lighteval_bench/.
Method
Growth is driven entirely by PGD — there is no separate schedule- or
cubic-driven growth mechanism (--gmp_tr_enabled=false,
--gmp_pruning_end_ratio=0.0). At every 8th step the mask's desired state is
the top-k-by-importance projection at the final target sparsity, and a
bisection accepts the largest swap set k whose measured self-KL displacement
stays within the trust region δ:
- Saliency: Fisher-weighted magnitude (
fisher_factor · w², Fisher diagonal from Adam'sexp_avg_sq), global scope across all masked linears. - Trust region: δ = 0.02, with the self-KL measured on a 4-sequence calibration batch drawn from the current rollout pool and refreshed every 32 steps.
- Asymmetric prune/revive: revive saturates at
min(k, revive_cand)while pruning continues tok, so accepted swaps are net-pruning until the target is met, after which the same rule degrades to pure maintenance. - Recovery objective: NTP + offline KD + on-policy KD, weighted 0.33/0.33/0.33. The on-policy term distills the dense teacher on the student's own rollouts, generated by an in-process vLLM engine (512 new tokens, temperature 0.6) and refreshed every 32 steps (64 refreshes over the run).
Growth is fast, recovery is long: the target 70% was reached at step 232 of 2048 (11.3%); the remaining 88.7% of training is recovery and mask maintenance.
Training configuration
| Base / teacher | Qwen/Qwen3-4B (dense, frozen) |
| Sparsity | 70% unstructured, global scope (final measured: 0.7000) |
| Steps | 2048 (batch 1 × grad-accum 8, seq len 8192) |
| LR | 1e-4, cosine, 256-step linear warmup |
| Mask / PGD cadence | mask_interval 32, pgd_interval 8 |
| Trust region δ | 0.02 (--gmp_pgd_kl_budget), calib size 4 |
| Saliency | fisher, Fisher EMA β = 0.999 |
| Loss weights | NTP 0.33 / KD 0.33 / on-policy KD 0.33 |
| Rollouts | in-process vLLM, 512 max new tokens, refresh every 32 steps |
| Data | ot3_fineweb_40k_qwen3_nostrip_8192.jsonl — 40k samples, 80/20 OT3-CoT / FineWeb mixture, 8192 tokens |
| Masking | hard mask (--gmp_ste=false): masked weights are re-zeroed after every optimizer step |
| Seed | 42 |
| Hardware | 1 × NVIDIA B200, ~5 h 40 min |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "cosmos1030/gmp-kd3e-1-s70pct-lr1e-4_20260901_080954"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
msgs = [{"role": "user", "content": "Solve: what is the 12th Fibonacci number?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[-1]:], skip_special_tokens=True))
Limitations
- No speedup on its own. Sparsity is unstructured and the weights are
stored as ordinary dense
safetensorscontaining zeros. Wall-clock or memory gains require sparse kernels or a structured (e.g. 2:4) variant; loading this model as-is costs the same as the dense base model. - Single seed, single run per cell. All ablation numbers above come from one run at seed 42. The source ablation package states its own resolution as roughly ±1 point, so differences smaller than that are not resolved — the B1 row in particular sits inside that band at other sparsities.
- Evaluation protocol. Numbers come from one
lightevalquick-profile configuration and are directly comparable within this ablation family; they are not necessarily comparable to numbers produced under other papers' harnesses, prompts, or sampling settings. - Inherits the capabilities, biases, and failure modes of
Qwen/Qwen3-4B. No dense-baseline row is included here, so the tables above quantify differences between pruning arms, not the cost of pruning relative to the dense model. Absolute code-generation ability is low at this sparsity (8.21 on LiveCodeBench).
Provenance
- Training log:
logs/queue_gmp_pgd_grow_to_target_4b/s70_klb0.02.log - W&B run:
r5j1uw8d(projectreasoning_qwen3_4b_nostrip8192) - Launcher:
b200_scripts/gmp_pgd_grow_to_target_qwen3_4b.sh 0.7 0.02 512 32 cosine 2048 1e-4 $OT3_DATA 8192 true reasoning_qwen3_4b_nostrip8192 fisher global 0.33,0.33,0.33 32 0 4 8 0.15 - Trained 2026-09-01, uploaded 2026-09-01 08:10 KST