JugnuLM-110M
A 109.7M parameter, deep-and-thin language model pretrained from scratch on ~8B tokens of FineWeb-Edu. It is the larger sibling of JugnuLM-53M and the baseline rung of an ongoing ablation study on what improves sub-150M models. Jugnu (जुगनू) = firefly.
Same deliberately conventional recipe as the 53M — Qwen3-style decoder (GQA, RoPE, SwiGLU, RMSNorm) with built-in QK-Norm and a z-loss — but scaled to ~110M with a deep-thin geometry (23 layers × 576, per MobileLLM's depth-over-width finding).
Results & the 53M → 110M jump
Evaluated with EleutherAI lm-evaluation-harness:
| Metric | JugnuLM-53M | JugnuLM-110M | Δ |
|---|---|---|---|
| BLiMP (acc) | 78.14 | 81.25 | +3.11 |
| ARC-Easy (acc) | 51.43 | 52.48 | +1.05 |
| WikiText-2 byte-perplexity ↓ | 2.04 | 1.95 | −0.09 |
The 81.25 BLiMP essentially matches GPT-X2-125M (81.28) — a top score on the sub-150M Tiny-ML Leaderboard — at ~12% fewer parameters and ~9× fewer training tokens (8B vs 75B). Depth + scale bought a clean, across-the-board gain over the 53M.
Reproduce:
lm_eval --model hf \
--model_args pretrained=altslate/JugnuLM-110M,dtype=bfloat16 \
--tasks blimp,arc_easy,wikitext --batch_size auto
Architecture
| Params | 109.7M |
| Base architecture | Qwen3 (Llama-style + built-in QK-Norm) |
| Layers | 23 (deep-thin) |
| Hidden size | 576 |
| Heads / KV heads | 9 / 3 (grouped-query attention) |
| Head dim | 64 |
| FFN intermediate | 1536 (SwiGLU) |
| Context | 2048 |
| Position / Norm | RoPE / RMSNorm + QK-Norm |
| Embeddings | tied |
| Tokenizer | SmolLM2 (49,152 vocab) |
Training
- Data:
HuggingFaceFW/fineweb-edu(sample-10BT), ~8B tokens seen, 2048-token blocks - Optimizer: AdamW (β 0.9/0.95, wd 0.1), grad clip 1.0
- Schedule: cosine, 2k warmup, peak LR 1.5e-3 → floor 1.5e-4
- Batch: ~0.5M tokens/step · Stability: QK-Norm + z-loss (1e-4)
- Hardware: 4× NVIDIA RTX PRO 4500 Blackwell, bf16, DDP + torch.compile
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/JugnuLM-110M")
model = AutoModelForCausalLM.from_pretrained("altslate/JugnuLM-110M")
ids = tok("The theory of relativity was developed by", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, repetition_penalty=1.3)[0], skip_special_tokens=True))
Intended use & limitations
Research / leaderboard base model — not instruction-tuned or aligned. Fluent English continuation and reasonable factual recall for its size; limited multi-step reasoning, occasional repetition, English-only. Not for production.
Citation / attribution
Trained by AltSlate Labs. Recipe and training code: https://github.com/AltSlate-Labs/jugnu