slm-125m — legal & financial base language model
A 125.8M-parameter Llama-style causal language model, pretrained from random weights on a corpus of US case law, SEC filings, and educational web text. This is a base text completer, not an instruction-tuned chat model — it continues text plausibly but was never trained to follow instructions or hold a conversation, and it has no factual grounding beyond patterns in its training data.
⚠️ Training status: partial. This checkpoint is the best validation perplexity checkpoint from a run that was stopped by its GPU spend cap at step 10,253 of a planned 17,645 (58% of a 5-epoch schedule), having seen 5.38B of a targeted 9.25B tokens. It is not the final planned checkpoint. Perplexity was still improving steadily when the run stopped (see the trajectory below), so a completed run would be expected to score better.
Model details
| Architecture | Llama (SwiGLU, RoPE, RMSNorm pre-norm, tied embeddings) |
| Parameters | 125,847,552 |
| Layers | 12 |
| Hidden size | 768 |
| Attention / KV heads | 12 / 12 |
| Intermediate size (SwiGLU) | 3,072 |
| Context length | 1,024 |
| Vocabulary | 16,384 (byte-level BPE, trained from scratch on this corpus) |
| RoPE theta | 10,000 |
| Precision trained in | bf16 autocast, fp32 master weights |
Training data
A 16,384-token byte-level BPE vocabulary and this model were both trained from scratch on a ~1.85B-token packed corpus (1,024-token windows), built from:
| Source | Share of train tokens | Dataset |
|---|---|---|
| US case law | 31.1% | HFforLegal/case-law (US split) |
| SEC filings | 43.9% | PleIAs/SEC |
| Educational web text | 25.0% | HuggingFaceFW/fineweb-edu (sample-10BT) |
The corpus was cleaned (boilerplate/OCR/repetition/language filters), deduplicated (MinHash near-dup removal on case law, exact-dup removal everywhere), and decontaminated against the CaseHOLD and LexGLUE evaluation splits before packing. 645,388 documents in total; validation is a disjoint 1% held-out slice (18.7M tokens).
Training recipe
| Setting | Value |
|---|---|
| Objective | Next-token cross-entropy |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 (excluded on 1-D params) |
| Peak / floor learning rate | 6e-4 / 6e-5 |
| Warmup | 200M tokens, then cosine decay |
| Precision | bf16 autocast |
| Gradient clipping | 1.0 global norm |
| Global batch | 524,288 tokens/optimizer step |
| Hardware | 8× H100, single-node DDP |
| Schedule (planned) | 5 epochs over the 1.85B-token corpus (~9.25B tokens, 17,645 steps) |
Validation perplexity trajectory
Full validation split, evaluated every 1,000 optimizer steps
(perplexity = exp(mean token-weighted cross-entropy)):
| Step | Val loss | Val perplexity |
|---|---|---|
| 1,000 | 2.777 | 16.07 |
| 2,000 | 2.524 | 12.48 |
| 3,000 | 2.421 | 11.26 |
| 4,000 | 2.361 | 10.60 |
| 5,000 | 2.319 | 10.17 |
| 6,000 | 2.286 | 9.83 |
| 7,000 | 2.257 | 9.55 |
| 8,000 | 2.236 | 9.35 |
| 9,000 | 2.214 | 9.15 |
| 10,000 | 2.194 | 8.97 |
| 10,253 (this checkpoint) | 2.188 | 8.92 |
Training loss fell from 9.87 (random init) to ~2.17 over the same span.
Sample completions
Generated with do_sample=True, temperature=0.8, top_p=0.95 at this
checkpoint. These are illustrative, not curated — a base model can
produce incoherent, biased, or fabricated legal/financial content, and
nothing it generates should be treated as legal or financial advice.
Prompt: "The court held that" Continuation: "...the defendant was not prejudiced by the admission of the testimony relating to the statement that he was not under arrest. In so holding, the court stated: We agree. In State v. Hensley..."
Prompt: "Net revenues for the fiscal year increased" Continuation: "...$2.5 million, or 7.1%, to $31.9 million in fiscal 1997 from $31.9 million in fiscal 1996. The increase in total revenues was attributable to the Company..."
Intended use & limitations
- Base completer only. No instruction tuning, no RLHF, no chat template
training —
<|user|>/<|assistant|>/<|system|>tokens are reserved in the tokenizer for a future instruction-tuning phase but this checkpoint was never trained on turn-structured data. - Not a source of legal or financial advice or fact. It will produce fluent, plausible-sounding legal and financial prose, including plausible case names, statutes, and figures that may be entirely fabricated.
- Partially trained, as noted above — expect rougher edges than a completed 5-epoch run.
- Small model, narrow domain. 125M parameters trained on ~1.85B tokens is small by current standards; general-knowledge and reasoning capability outside the legal/financial/web-text domain is limited.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "karuppusamym/slm-125m-legal-financial"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
inputs = tok("The court held that", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))
License
Model weights released under Apache 2.0. Training data provenance: US case law is public-domain court opinion text; SEC filings are public regulatory filings; FineWeb-Edu is a permissively-licensed educational web-text dataset. This repository does not redistribute the training data itself.