slm-125m-base
A 125.8M-parameter Llama-shaped language model pretrained from scratch on a legal-first corpus (US case law + SEC filings + a slice of FineWeb-Edu). Built with a custom 16,384-token BPE tokenizer.
This is a base / foundation model -- it does next-token continuation, not instruction following or chat. Expect rough, domain-flavored completions; it was trained on a small budget (step 19,334, ~10.14B tokens seen).
Architecture
| Params | ~125.8M (tied embeddings) |
| Layers | 12 |
| Hidden size | 768 |
| Heads / KV heads | 12 / 12 (MHA) |
| Context length | 1,024 |
| Vocab | 16,384 (custom BPE) |
| Activation | SwiGLU (silu) |
| Position | RoPE (theta 10000) |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("analyticspro/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("analyticspro/slm-125m-base")
model.eval()
ids = tok("The court held that", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=120, do_sample=True,
temperature=0.8, top_p=0.95,
pad_token_id=tok.eos_token_id)
print(tok.decode(out[0], skip_special_tokens=True))
Limitations
Small model, small pretraining budget, and a legal-heavy corpus: outputs can be factually wrong, repetitive, or biased toward legal/regulatory phrasing. Not suitable for production or any high-stakes use. For research and demos only.