slm-125m-base
A 125M-parameter Llama-style base language model pretrained from scratch — tokenizer, data pipeline, and training loop included — on a legal/financial corpus (US case law + SEC filings + a web slice, ~2.19B tokens).
🎮 Try it in your browser (WebGPU, no install): slm-125m-playground.vercel.app — the int8 ONNX export runs fully client-side via 🤗 Transformers.js (~139 MB one-time download, then cached/offline).
| Parameters | ≈125.8M (tied embeddings) |
| Training tokens | ~2.19B (≈17.5 tokens/param — approximately Chinchilla-optimal) |
| Held-out validation perplexity | 11.01 |
| Context length | 1,024 tokens |
| Precision | Trained in bf16 · int8 (q8) ONNX export for browser |
This is a base completer, not a chat model. Prompt it with the start of a sentence and it continues fluently in the legal/financial register. It does not answer questions, follow instructions, or reliably state facts — at this size, world knowledge is capped at roughly ~2 bits/param. Instruction-tuned (SFT) and preference-tuned variants are on the roadmap below.
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("genaiquest/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("genaiquest/slm-125m-base")
ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True,
temperature=0.8, min_new_tokens=40, repetition_penalty=1.3)[0]))
Suggested sampling: temperature 0.8 · top_p 0.95 · repetition_penalty 1.3 (small base models loop without a repetition penalty).
Prompts that work well (in-register continuations):
IN THE UNITED STATES DISTRICT COURTPursuant to Section 10(b) of the SecuritiesThe Company's revenue for the fiscal yearThe court finds that the evidence presented
Architecture
Deliberately vanilla geometry, mapping 1:1 to transformers.LlamaConfig — so it exports to ONNX (and GGUF) with zero custom-converter work. That design choice is exactly what makes the in-browser playground possible.
| Component | Spec |
|---|---|
| Family | Llama-style decoder-only |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 (head dim 64) |
| KV heads | 12 — MHA, num_kv == num_heads → export-safe |
| MLP | SwiGLU, intermediate 3072 (4×), SiLU gate |
| Normalization | RMSNorm (eps 1e-5), pre-norm |
| Positional encoding | RoPE, θ = 10,000 |
| Vocabulary | 16,384 |
| Embeddings | Tied (input embedding == LM head) |
| Attention bias | False |
Parameter budget (tied):
embed 16,384 × 768 = 12.58M
attn/layer 4 × 768² = 2.36M
mlp/layer 3 × 768 × 3072 = 7.08M
(attn + mlp) × 12 layers = 113.2M
total ≈ 125.8M
Training
- Data: US case law + SEC filings + a general web slice, deduplicated; ~2.19B tokens total. Case law and SEC filings are public-domain US government works. [40%-40%-20%]
- Tokenizer: 16,384-token BPE trained from scratch on the corpus.
- Compute: 1 epoch on 8×H100. [FILL: wall-clock hours, tokens/sec, approx cost — e.g., "X hrs ≈ $Y at market spot rates"]
- Recipe: optimizer (AdamW), peak LR, schedule (cosine), 200M token warmup, global batch size 32, 0.5M tokens/step, grad clip, weight decay
- Scaling note: 2.19B tokens ÷ 125.8M params ≈ 17.5 tokens/param, close to the Chinchilla compute-optimal ratio (~20) — this run prioritized compute-optimality over deliberate overtraining.
Loss curve
Evaluation
| Metric | Value |
|---|---|
| Validation perplexity (held-out, in-domain) | 11.01 |
Perplexity is measured on a held-out slice of the training distribution (legal/financial register). Expect materially higher perplexity on out-of-domain text; this model is a domain specialist by construction.
Intended use & limitations
Intended: studying small-model pretraining end to end; domain-register text continuation; a base checkpoint for SFT/DPO/RAG experiments; edge/browser inference demos; teaching material for the full train→quantize→deploy pipeline.
Not intended: legal or financial advice, factual question answering, chat, or any production decision-making. Outputs can be fluent and wrong — at 125M parameters that is a certainty, not a risk. Generated text may echo the style of court filings and SEC documents; verify nothing here as fact.
ONNX / browser deployment
The repo includes an int8 (q8) ONNX export consumed by 🤗 Transformers.js in the playground. Because the architecture maps 1:1 to LlamaConfig, the export pipeline is: bf16 safetensors → ONNX → dynamic int8 quantization, with no custom conversion code.
Roadmap
This is checkpoint 1 of a series building the full enterprise-SLM stack in public:
- ✅ slm-125m-base — pretraining from scratch (this model)
- ⏳ -sft — supervised fine-tuning for instruction following / QnA
- ⏳ -dpo/ppo — preference tuning
- ⏳ RAG-grounded variants for document QnA with citations
- ⏳ Serving optimization study on this model family (vLLM, CUDA graphs, FP8) — companion to my vLLM H100 serving experiments
Author
Bhabani Nayak — AI engineering leader; agentic AI at consumer scale (Sephora; previously Shopify, Twitter, Apple). IEEE Senior Member. LinkedIn · Substack · Playground
Citation
@misc{nayak2026slm125m,
author = {Nayak, Bhabani},
title = {slm-125m-base: a 125M Llama-style language model pretrained from scratch on legal/financial text},
year = {2026},
url = {https://huggingface.co/genaiquest/slm-125m-base}
}