MiMo-7B GDN Hybrid — stage3-2.0B-OPD
Decayed final of the 2.0B-token rung of the MiMo on-policy-distillation (OPD)
WSD ladder (1:4 uniform hybrid of Qwen2 attention and Gated DeltaNet layers,
teacher XiaomiMiMo/MiMo-7B-RL-0530). Run
mimo-mathmix-h32k-runpod-2p0b-v1-20260905; snapshots/final, step 12887,
2,000,126,188 consumed generation tokens; model window 65,536.
Recipe: extends arianraje/mimo-7b-gdn-hybrid-1.8B-OPD from its pre-decay
state (1,721.4M tokens) with no recipe change — the same stage-3 math
mixture, rollout horizon 32,768, sampler window 49,152, 600-step linear decay,
queue factor 6.0. This is the ladder's first matched-recipe A/B at H=32K, so
differences vs the 1.8B row are a pure tokens-scaling read (the 1.8B row's own
confounds vs earlier rows carry forward unchanged). The pre-decay trunk of this
rung is public at arianraje/mimo-7b-gdn-opd-predecay-1921m-step12391.
Evaluation (TABLES.md protocol, raw JSONs under full_eval/)
| Metric | 1.8B-OPD | 2.0B-OPD |
|---|---|---|
| AIME24 think pass@1 / pass@8 @32K (n=30×8) | 63.3 / 76.7 | 59.6 / 80.0 |
| AIME25 think pass@1 / pass@8 @32K (n=30×8) | 49.2 / 73.3 | 47.1 / 73.3 |
| MATH-500 think pass@1 @32K | 93.4 | 93.8 |
| MATH-500 no-think pass@1 @4K | 70.2 | 70.8 |
| GSM8K no-think strict / flexible @1K | 58.5 / 65.2 | 58.1 / 62.2 |
| MMLU 5-shot | 53.8 | 54.6 |
| PIQA / HellaSwag / ARC-E / ARC-C / Winogrande | 72.0 / 60.3 / 62.8 / 39.6 / 59.7 | 72.5 / 59.9 / 62.7 / 39.5 / 58.8 |
| NIAH multikey 4K / 8K / 16K / 32K (n=500) | 96.8 / 97.2 / 95.0 / 84.6 | 96.8 / 96.8 / 94.8 / 83.8 |
| NIAH single, multiquery (all lengths) | ≥99.8 | ≥100.0 |
| trunc / mean gen tokens: MATH-500, AIME24, AIME25 | 3.6%/8168, 26.7%/21925, 32.5%/23552 | 3.8%/7930, 29.2%/21862, 38.3%/24357 |
AIME cells are n=30 problems (±~14 pts on pass@1) — trend data; MATH-500
(n=500) is the powered cell. full_eval/*.generations.json carry every sample.
Loading
Custom architecture; needs the mimo_gdn model registration from the
training repo (transformers 4.57.x + flash-linear-attention 0.5.x), same as
the other arianraje/mimo-7b-gdn-hybrid-*-OPD checkpoints.