JugnuLM-110M-R3 (value residuals + Muon + data blend)
What "R3" means. This is rung 3 of the JugnuLM phase-2 ablation ladder. We take rung R2 — JugnuLM-110M-R2 (value residuals + Muon) — and change one thing: the training data. Instead of single-source FineWeb-Edu, R3 trains on a blend (per-sequence weighted mix), holding architecture, optimizer, schedule, tokens, batch, and seed fixed.
What changed: the data blend
| source | weight | role |
|---|---|---|
| FineWeb-Edu (sample-10BT) | 55% | language + general knowledge |
| DCLM-baseline | 35% | diversity → broader distribution |
| FineMath-4plus | 10% | light math reasoning |
Sequences are drawn per-step from a weighted-random source (a shuffled blend, not a staged curriculum). Validation stayed a fixed FineWeb-Edu held-out set for comparable curves.
Result — an honest negative for ARC
| 110M rung | BLiMP ↑ | ARC-Easy ↑ | WikiText-2 byte-ppl ↓ |
|---|---|---|---|
| R0 — baseline | 81.25 | 52.48 | 1.95 |
| R1 — + value residuals | 81.10 | 54.67 | 1.94 |
| R2 — + Muon | 80.78 | 56.10 | 1.932 |
| R3 — + data blend | 81.79 | 53.62 | 1.9092 |
R3 has the best BLiMP and best perplexity of any rung — but ARC-Easy dropped −2.48 vs R2 (56.10 → 53.62). Since ARC-Easy is our binding constraint, R3 is not kept in the ladder (it does not beat R2 on the metric that gates leaderboard rank; its EFF is marginally below R2's). We publish it anyway, as the ablation result and as our strongest grammar/perplexity checkpoint.
Why the blend helped BLiMP/perplexity but hurt ARC
FineWeb-Edu's educational-quality filtering is closely aligned with ARC-Easy's
grade-school science questions. Diluting it 100% → 55% with general web text (DCLM) and math
(FineMath) improved broad language modeling (grammar, perplexity) but removed ARC-relevant
educational signal — DCLM is diverse but less science-dense, and FineMath targets math
reasoning, not ARC science. The lesson: at this scale, for ARC-Easy the educational
distribution matters more than raw diversity. Evaluated with EleutherAI lm-evaluation-harness.
⚠️ Loading — requires trust_remote_code
Value residuals are a custom attention pathway, so this model must be loaded with the included modeling file. Loading it as a stock Qwen3 silently drops the residual and understates quality.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M-R3")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M-R3", trust_remote_code=True)
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
Architecture & training
Identical architecture/optimizer to R2 (value residuals + Muon): Qwen3-arch, 23 layers × 576, GQA 9/3, RoPE, SwiGLU, RMSNorm, QK-Norm, tied embeddings, SmolLM2 tokenizer (49,152), z-loss 1e-4. Trained from scratch on ~8.4B tokens of the 55/35/10 blend above, Muon (peak 2e-2) + AdamW (1.5e-3) on a shared cosine, ~0.5M-token global batch, bf16, DDP on 4× RTX PRO 4500 Blackwell. Only the data differs from R2.
Intended use & limitations
Research base model (not instruction-tuned). Best-in-family English fluency/grammar and lowest perplexity, but weaker ARC-Easy reasoning than R1/R2. Occasional repetition, English-only. Not for production.
Links
- Previous rung (best ARC): altslate/JugnuLM-110M-R2
- Baseline: altslate/JugnuLM-110M
- Training code & ablation ladder: https://github.com/AltSlate-Labs/jugnu