slm-125m-legal-base

A 125M-parameter Llama-architecture base model pretrained from scratch on a legal- and finance-heavy corpus (US case law, SEC filings, and a slice of FineWeb-Edu).

This is a base model: it does next-token prediction only. It has not been instruction-tuned, chat-tuned, or RLHF'd, and it will not follow instructions.

Model details

The tokenizer is custom and trained on this corpus. A 16K vocab is small by modern standards; it was chosen to keep the embedding table from dominating a 125M budget.

Independent absolute score (2026-08-15)

Scored by a Claude Sonnet judge on four axes out of 10, over 300 held-out grounded Q&A prompts, on the same scale as every other checkpoint in this project.

This is the expected result and not a defect. A pretrained base model has no instruction tuning: prompted as a plain continuation (it has no chat template), 93% of its outputs trip a repetition detector, averaging ~1,175 characters where the SFT model answers in ~180. It continues text; it does not answer questions. The number is here so the fine-tuning step has a measured floor to stand on.

Training data

Pretraining used 2,039,072,768 tokens across 1,991,282 packed 1024-token windows, with a 1% held-out validation split (20,601,856 tokens / 20,119 windows).

The mix is legal-first rather than the more common 70/20/10 web-heavy split: the two legal sources cap at roughly 2B tokens, so the pipeline takes all of both and caps web at 0.5B.

Data preparation: a 6-step deterministic cleaning chain (line filtering, non-alphanumeric ratio, minimum document length, repetition/n-gram filtering, language detection, and an OCR-quality gate applied to case law only), then MinHash near-duplicate removal, then 13-gram decontamination against LexGLUE / CaseHOLD evaluation sets (480,908 eval 13-grams).

Training procedure

Evaluation

Perplexity over the entire held-out validation split (20,558,208 target tokens):

This is in-domain perplexity on data drawn from the same mix as training. It is not a capability benchmark, and it is not comparable to perplexity numbers computed against a different corpus or with a different tokenizer — a 16K vocab makes perplexity look lower than it would with a 32K or 50K vocab.

No downstream task benchmarks (MMLU, LegalBench, CaseHOLD, etc.) have been run.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-base")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-base")

ids = tok("The plaintiff shall bear the burden of", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=60, do_sample=True,
                     temperature=0.8, top_p=0.95,
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0], skip_special_tokens=True))

Limitations and intended use

This model is a research and educational artifact — a demonstration that a coherent domain-specific base model can be pretrained end-to-end on a small budget. It is not suitable for production use, and specifically:

• It does not do arithmetic. It will confidently produce a revenue figure, a comparison figure, and a percentage change that do not follow from each other. This is expected at 125M parameters. Do not treat any number it generates as a computation.
• It is not a source of legal or financial fact. It reproduces the register of judicial opinions and SEC filings fluently, which makes fabricated citations, case names, docket numbers, and financial figures look plausible. It hallucinates freely.
• It is not legal or financial advice, and must not be used as a substitute for a qualified professional, nor deployed in any advisory, compliance, or decision-making capacity.
• It does not follow instructions. It is a base model; prompt it as a text continuation model.
• 1024-token context, so it cannot process a full filing or opinion.
• It inherits the biases of its sources. US case law spans a long historical range and encodes the prejudices of the courts and eras that produced it; SEC filings are corporate self-representation. The model reflects both.
• English only, US-jurisdiction-skewed.

License and attribution

The model weights are released under Apache-2.0.

The training corpus draws on three sources with their own terms. None carries a share-alike or non-commercial clause, so nothing upstream constrains the license of these weights, but two require attribution:

This section is that attribution, and it stands whether or not those terms legally reach model weights — a question that is genuinely unsettled and largely untested in court. The common position is that weights are not a derivative work of the training data; Apache-2.0 plus visible credit is the choice that holds up under either answer.

None of this is legal advice. If you intend to use this model commercially, get your own counsel to look at it.

Training pipeline

The full pipeline — streaming, cleaning, deduplication, decontamination, tokenizer training, packing, and distributed pretraining — runs on Modal and is reproducible. Data preparation (Phases 0-4) cost $1.73 on CPU; pretraining cost ~$11.40 on 8x H100.