MiMo-7B GDN Hybrid — stage3-1.6B-OPD
Decayed final of the 1.6B-token rung of the MiMo on-policy-distillation (OPD)
WSD ladder (1:4 uniform hybrid of Qwen2 attention and Gated DeltaNet layers,
teacher XiaomiMiMo/MiMo-7B-RL-0530). Run
mimo-mathmix-h16k-runpod-1p6b-v1-20260901; snapshots/final, step 10404,
1,600,093,536 consumed generation tokens; rollout horizon H=16,384 (evaluated
at a 32,768-token cap); model window 65,536.
Recipe: extends arianraje/mimo-7b-gdn-hybrid-1.4B-OPD from its pre-decay
state (1,336.9M tokens) on the byte-identical recipe (stage-3 math mixture,
max_queue_factor 6.0) with ONE change: decay_steps 482 → 600. The
pre-decay trunk state of this rung is public at
arianraje/mimo-7b-gdn-opd-predecay-1521m-step9888.
Evaluation (TABLES.md protocol, raw JSONs under full_eval/)
| Metric | 1.4B-OPD | 1.6B-OPD |
|---|---|---|
| AIME24 think pass@1 / pass@8 @32K (n=30×8) | 62.1 / 83.3 | 62.1 / 83.3 |
| AIME25 think pass@1 / pass@8 @32K (n=30×8) | 39.2 / 63.3 | 47.5 / 73.3 |
| MATH-500 think pass@1 @32K | 95.4 | 93.8 |
| MATH-500 no-think pass@1 @4K | 69.2 | 68.2 |
| GSM8K no-think strict / flexible @1K | 51.6 / 60.3 | 57.6 / 63.8 |
| MMLU 5-shot | 54.0 | 54.2 |
| PIQA / HellaSwag / ARC-E / ARC-C / Winogrande | 73.2 / 60.4 / 64.1 / 39.8 / 59.0 | 72.6 / 60.3 / 63.6 / 39.1 / 59.3 |
| NIAH multikey 4K / 8K / 16K / 32K (n=500) | 97.4 / 97.2 / 93.0 / 83.4 | 98.6 / 98.0 / 96.6 / 89.4 |
| NIAH single, multiquery (all lengths) | ≥99.8 | ≥99.8 |
| trunc / mean gen tokens: MATH-500, AIME24, AIME25 | 2.2%/7807, 27.1%/21515, 36.7%/24187 | 3.4%/8169, 30.4%/22273, 35.8%/23806 |
AIME cells are n=30 problems (±~14 pts on pass@1) — trend data; MATH-500
(n=500) is the powered cell. full_eval/*.generations.json carry every sample.
Loading
Custom architecture; needs the mimo_gdn model registration from the
training repo (transformers 4.57.x + flash-linear-attention 0.5.x), same as
the other arianraje/mimo-7b-gdn-hybrid-*-OPD checkpoints.