SLM125mLIVE base (125M)
A 125.8M-parameter Llama-architecture base language model pretrained from scratch on US court opinions, SEC filings, and educational web text. It is a raw pretrained model: it is not instruction-tuned or aligned, and it is not a source of legal or financial advice.
Model
| Architecture | LlamaForCausalLM, 12 layers, hidden 768, 12 heads (MHA), SwiGLU 3072, RoPE θ=10000, RMSNorm |
| Parameters | 125,847,552 (tied input/output embeddings) |
| Context length | 1024 tokens |
| Tokenizer | byte-level BPE, vocab 16,384, trained on this corpus |
| Special tokens | <|bos|>=0, <|eos|>=1, <|pad|>=2, <|unk|>=3 (plus unused chat tokens) |
Documents were packed into 1,024-token windows separated by <|eos|>; no BOS token was used in
training, so prompts work best when prefixed with <|eos|>.
Training data
2.04B unique training tokens (plus a 20.6M-token held-out validation split: every 100th window).
| Source | Documents | Train tokens | Share |
|---|---|---|---|
| HFforLegal/case-law | 206,684 | 715M | 35% |
| PleIAs/SEC | 45,035 | 860M | 42% |
| HuggingFaceFW/fineweb-edu | 418,405 | 464M | 23% |
Processing: line-level filtering, boilerplate stripping, and length, repetition, language, and OCR-garble filters. Then MinHash near-duplicate removal (case law) and exact-duplicate removal. Documents sharing a 13-gram with the CaseHOLD test set were removed (24,177 documents) so the CaseHOLD result below is not contaminated.
Training
| Tokens seen | 6.12B (11,667 steps, 3.0 epochs) |
| Batch | 524,288 tokens (512 × 1024) |
| Optimizer | AdamW (β1=0.9, β2=0.95, wd=0.1), grad clip 1.0 |
| LR schedule | 200M-token warmup to 0.0006, cosine to 6e-05 |
| Precision | bf16 autocast, torch.compile, DDP |
| Hardware | 8× NVIDIA H100 on Modal, 2.26M tokens/s |
Evaluation
Held-out validation (all 20.6M validation tokens):
| Source | Loss | Perplexity |
|---|---|---|
| case-law | 2.189 | 8.92 |
| sec | 1.599 | 4.95 |
| fineweb-edu | 3.105 | 22.31 |
| All | 2.149 | 8.57 |
Validation loss during training (2,048-window subset):
| Step | Val loss | Perplexity |
|---|---|---|
| 1,000 | 2.784 | 16.19 |
| 2,000 | 2.530 | 12.55 |
| 3,000 | 2.424 | 11.29 |
| 4,000 | 2.361 | 10.60 |
| 5,000 | 2.317 | 10.14 |
| 6,000 | 2.282 | 9.79 |
| 7,000 | 2.251 | 9.49 |
| 8,000 | 2.227 | 9.27 |
| 9,000 | 2.208 | 9.10 |
| 10,000 | 2.193 | 8.97 |
| 11,000 | 2.183 | 8.87 |
| 11,667 | 2.179 | 8.84 |
CaseHOLD (LexGLUE case_hold test, 3,600 examples, 5-way multiple choice, zero-shot).
Each candidate holding is scored by its log-likelihood as the continuation of the citing text that
precedes it (lm-eval-harness style):
| Metric | Accuracy |
|---|---|
| acc (log-likelihood) | 12.0% |
| acc_norm (per-byte log-likelihood) | 17.7% |
| Chance | 20% |
The model does not solve CaseHOLD zero-shot. acc matches the "always pick the shortest holding" baseline (11.4%): correct holdings are longer on average (168 vs 152 characters), and summed log-likelihood favors shorter text. acc_norm removes most of that length effect but stays below chance. A lexical baseline (pick the holding that shares the most words with the context) reaches 35.0%. Treat CaseHOLD as a fine-tuning target for this model, not a zero-shot score.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nagbhaskar55/slm125mlive-base")
model = AutoModelForCausalLM.from_pretrained("nagbhaskar55/slm125mlive-base")
ids = tok("<|eos|>The court held that", return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))
Limitations
- Small base model: it produces fluent-looking but frequently incorrect or invented text, including invented case names, citations, statutes, and financial figures.
- Domain skew: US court opinions (some OCR noise) and SEC filings dominate; general knowledge is limited.
- No safety tuning, instruction tuning, or RLHF.
- License:
other. Check the licenses and terms of the source datasets before any use.
Files
metrics.jsonl: training log.training_summary.json: run summary.eval_results.json: full evaluation output, including sampled continuations.