slm-500m-base
A 517.8M-parameter Llama-style decoder, pretrained from scratch (fresh weights, fresh tokenizer) on a legal/financial corpus. This is a base model: it completes text, it does not answer questions.
Architecture
| parameters | 517,795,840 |
| layers | 24 |
| hidden size | 1280 |
| attention heads | 20 (head dim 64, MHA) |
| MLP | SwiGLU, inner 3456 |
| norm | RMSNorm, pre-norm |
| positions | RoPE (theta 10000) |
| context length | 1024 |
| vocab | 32,768 (byte-level BPE, trained on this corpus) |
| embeddings | tied |
| precision | bf16 |
Training
- Data: thesreedath/slm-pretraining-corpus (~2.19B unique tokens; US case law, SEC filings, fineweb-edu), cleaned, MinHash-deduplicated, and decontaminated against CaseHOLD/LexGLUE.
- Recipe: AdamW (0.9, 0.95), wd 0.1, LR 0.0003 -> 3e-05 cosine, 200M-token warmup, global batch 524,288 tokens, grad clip 1.0.
- Hardware: 8x B200, DDP, torch.compile.
- Epochs: n/a | tokens seen: 0.0B
Results
| metric | value |
|---|---|
| validation perplexity | 7.912 |
| bits per byte | n/a |
Read bits-per-byte, not perplexity, when comparing this to a model with a different tokenizer. Perplexity is per-token: a larger vocabulary packs more text into each token, making each prediction harder and the perplexity higher, for a model that is doing better. We measured exactly that during this run: at step 3000 this model's perplexity (11.51) looked worse than the 125M's (11.33) while its bits-per-byte (0.694) was 3.9% better. Bits-per-byte is tokenizer-invariant and is the honest cross-model number.
What this model is not
A base model is a completer. Give it the start of a passage and it continues in-register. It does not follow instructions, and it does not reliably know facts: at this scale, knowledge capacity is roughly 2 bits per parameter. For question answering, use the fine-tuned variants; for factual reliability, use retrieval.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("thesreedath/slm-500m-base")
model = AutoModelForCausalLM.from_pretrained("thesreedath/slm-500m-base")
ids = tok("The plaintiff alleges that the defendant", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True, top_k=50,
temperature=0.8, min_new_tokens=40)
print(tok.decode(out[0], skip_special_tokens=True))
min_new_tokens matters: a base model will otherwise sometimes emit EOS
immediately.