SLM-300M
A 304M-parameter decoder-only language model trained from scratch on 6.08B tokens of FineWeb-Edu, the Chinchilla-optimal budget for its size, on a single A100. Everything here was built from scratch: the tokenizer, model, data pipeline, Muon optimizer and the fault-tolerant training system. Code: github.com/Darpan-Maurya/SLM.
On zero-shot benchmarks it beats OPT-350M and BLOOM-560M and comes within a point of Pythia-410M on average, with 30-60x less training data, and leads all of them on ARC-Easy.
This is a base model: it continues text and has not been instruction-tuned or safety-tuned.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Drm39/slm-300m"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16)
inputs = tokenizer("Photosynthesis is the process by which", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80, do_sample=True,
temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(tokenizer.decode(out[0], skip_special_tokens=True))
trust_remote_code=True loads modeling_slm.py from this repository: the architecture (per-head
QK-norm together with NoPE layers) is not one that transformers ships. That file reproduces the
training implementation exactly: logits are bit-identical, and greedy decoding agrees token for token
with and without the KV cache and in left-padded batches. Tested with transformers 5.18.
Model
| Parameters | 304.1M (270.6M non-embedding), embeddings tied |
| Layers / width | 24 layers, d_model 1024, SwiGLU FFN 2816 |
| Attention | 16 query heads, 4 KV heads (GQA), head dim 64, RMSNorm on queries and keys |
| Positions | RoPE (theta 10,000) in 3 of every 4 layers; every 4th layer is NoPE |
| Normalisation | pre-norm RMSNorm, statistics in fp32 |
| Context | 2,048 tokens, trained with intra-document attention masking |
| Tokenizer | byte-level BPE trained on FineWeb-Edu, 32,768 tokens, GPT-4 split pattern |
Training
| Data | FineWeb-Edu sample-10BT, files 000-007: 5.45B tokens (about 1.1 epochs; the second pass is reshuffled) |
| Tokens | 6.08B = 11,596 steps x 524,288 tokens (256 sequences of 2,048) |
| Optimiser | Muon (lr 0.02, momentum 0.95) on the 96 hidden matrices; AdamW (lr 6e-4, betas 0.9/0.95, wd 0.1) on embeddings and gains |
| Schedule | warmup-stable-decay: 1,000 warmup steps, constant, then a 1-sqrt decay over the last 20% to 10% of peak |
| Regularisation | z-loss 1e-4, gradient clipping 1.0, no dropout |
| Precision | bf16 autocast, fp32 master weights, torch.compile, FlexAttention document masks |
| Hardware | one A100 40GB on Google Colab, about 26 GPU-hours; 62-69k tokens/s at 48-54% MFU |
| Final validation | loss 2.86 nats/token, 0.897 bits per byte |
Checkpoints went off the machine every ~10 minutes, to two Kaggle datasets that take turns so one complete copy always survives a failed upload; resumes are exact (data order, RNG, optimiser state, schedule) and every file is sha256-verified before it is loaded.
Evaluation
Zero-shot, scored by the log-likelihood of each answer continuation on the full evaluation sets
(10,042 HellaSwag, 2,376 ARC-Easy, 1,172 ARC-Challenge, 1,838 PIQA, 1,267 WinoGrande, 500 OpenBookQA).
The metric per task follows the usual lm-eval convention (acc or length-normalised acc_norm).
| Model | Training tokens | ARC-E | ARC-C | PIQA | HellaSwag | OBQA | WinoGrande | Avg |
|---|---|---|---|---|---|---|---|---|
| SLM-300M | 6B | 56.6 | 27.5 | 65.2 | 36.5 | 31.6 | 51.7 | 44.9 |
| OPT-350M | 180B | 41.9 | 25.7 | 64.8 | 36.2 | 33.3 | 52.4 | 42.4 |
| BLOOM-560M | 366B | 43.7 | 27.5 | 65.1 | 36.5 | 32.6 | 52.2 | 42.9 |
| Pythia-410M | 300B | 47.1 | 30.3 | 67.2 | 40.1 | 36.2 | 53.4 | 45.7 |
| MobileLLM-350M | 1T | 53.8 | 33.5 | 68.6 | 49.6 | 40.0 | 57.6 | 50.5 |
LAMBADA (OpenAI version, exact greedy last word): 32.3%.
Baseline numbers are from Table 3 of the MobileLLM paper; they come from a different evaluation setup, so treat differences of a couple of points as noise. The pattern is expected from the data: an educational corpus helps knowledge questions (ARC) most and everyday narrative completion (HellaSwag) least, where sheer token count matters more.
Limitations
- A base model: it completes text and does not follow instructions.
- Small and briefly trained: fluent and on-topic, but often factually wrong, for example "Plants do not need sunlight to grow."
- English only; context limited to 2,048 tokens.
- Trained on filtered web text and not safety-tuned, so it can reproduce biases in that data.
Citation
@misc{maurya2026slm300m,
title = {SLM-300M: a small language model trained from scratch},
author = {Darpan Maurya},
year = {2026},
url = {https://huggingface.co/Drm39/slm-300m}
}
Built on FineWeb-Edu (Penedo et al., 2024) and the Muon optimiser (Jordan et al., 2024).