slm125m-live
A 125M-parameter Llama-architecture language model pretrained from scratch on a legal- and finance-heavy corpus. Trained end to end on Modal; the tokenizer is also trained from scratch on this corpus (16,384-token byte-level BPE).
This is a base model. It has had no instruction tuning, no RLHF, and no safety alignment. At 125M parameters it will confabulate freely — it is a research and teaching artifact, not a source of legal or financial advice.
Architecture
| Parameters | 125,847,552 (125.8M) |
| Layers / hidden / heads | 12 / 768 / 12 (MHA) |
| Context length | 1024 |
| Vocab | 16,384 (byte-level BPE, trained on this corpus) |
| Position encoding | RoPE (theta 10000) |
| Activation / norm | SwiGLU / RMSNorm |
| Tied embeddings | yes |
Training data (2.04B unique tokens)
| Source | Tokens | Share |
|---|---|---|
| case-law | 716M | 35.1% |
| sec | 860M | 42.2% |
| fineweb-edu | 465M | 22.8% |
Built from HFforLegal/case-law (US court opinions), PleIAs/SEC (SEC filings) and
HuggingFaceFW/fineweb-edu (sample-10BT, general fluency filler). The legal sources
are the binding constraint: together they hold only ~2B clean tokens, so the mix is
"take all the legal text, add a small web slice" rather than a chosen ratio.
Pipeline: stream -> 6-step deterministic clean (line filters, boilerplate strip, 4-gram repetition, ASCII/langdetect English gate, dictionary-based OCR gate on case-law) -> MinHash near-dedup + exact dedup -> 13-gram decontamination against CaseHOLD/LexGLUE -> pack into 1024-token windows, 99/1 train/val split.
Training
| Tokens seen | 8.16B (4 epochs) |
| Steps | 15,568 |
| Global batch | 524,288 tokens |
| Optimizer | AdamW (betas 0.9/0.95, wd 0.1, clip 1.0) |
| LR schedule | cosine 0.0006 -> 6e-05, 200M warmup tokens |
| Precision | bf16 autocast, fp32 master weights |
| Hardware | 8x NVIDIA H100 (DDP, single node) |
| Final val loss | 2.1228 |
Evaluation (held-out 1% split)
| Split | Perplexity |
|---|---|
| ALL | 8.31 |
| case-law | 8.68 |
| sec | 4.80 |
| fineweb-edu | 21.61 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("AnandHaridas1980/slm125m-live")
model = AutoModelForCausalLM.from_pretrained("AnandHaridas1980/slm125m-live")
ids = tok("The plaintiff filed a motion to dismiss on the grounds that", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=True, top_p=0.95)[0]))
Limitations
Base model, English only, 1024-token context. The case-law source is OCR'd and retains some scanning noise despite the dictionary gate. Training data is skewed to older SEC filings. Do not use for legal or financial advice.