Sarath569/slm-125m-legal-base
A 125.8M-parameter Llama-style decoder-only language model pretrained from scratch on a cleaned, deduplicated, decontaminated corpus of US case law, SEC filings, and educational web text (~2.04B unique tokens; legal-first mix).
Model
| Architecture | Llama (RoPE, SwiGLU, RMSNorm, tied embeddings) |
| Parameters | 125,848,320 |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context length | 1024 |
| Vocab (byte-level BPE) | 16,384 |
| Precision | bf16 |
Training
| Tokens seen | 4.08B (2 epochs over 2.04B) |
| Optimizer steps | 7,778 |
| Optimizer | AdamW (β=0.9/0.95, wd=0.1, grad-clip=1.0) |
| LR schedule | cosine 6e-4 → 6e-5, 200M-token warmup |
| Hardware | 8× H100 (data-parallel) |
| Final val loss / ppl | 2.3307 / 10.29 |
Data mix (realized)
- case-law (
HFforLegal/case-law) — ~722M tokens (35%) - sec (
PleIAs/SEC) — ~869M tokens (42%) - fineweb-edu (
HuggingFaceFW/fineweb-edu) — ~469M tokens (23%)
Decontaminated against CaseHOLD / LexGLUE (case_hold) with 13-gram overlap filtering.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("Sarath569/slm-125m-legal-base")
model = AutoModelForCausalLM.from_pretrained("Sarath569/slm-125m-legal-base", torch_dtype=torch.bfloat16)
ids = tok("The plaintiff shall bear the burden of", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, top_k=50, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
Limitations
This is a small (~126M) base model, not instruction-tuned or aligned. It will produce
fluent but unreliable text, can hallucinate legal/financial facts, and must not be used
for legal or financial advice. Reserved chat tokens (<|user|>, <|assistant|>, <|system|>)
are present in the tokenizer for downstream fine-tuning.