DexterP29/slm125LIVE-base
A 126M-parameter Llama-architecture base model pretrained from scratch on a legal/financial corpus. Not instruction-tuned.
Model
| Parameters | 125,847,552 |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context length | 1024 |
| Vocabulary | 16384 (byte-level BPE trained on this corpus) |
| Tied embeddings | yes |
Training data
2.04B tokens, deduplicated (MinHash/LSH + exact hash) and decontaminated against the CaseHOLD benchmark via 13-gram overlap:
- sec — 861M tokens (42%)
- case-law — 714M tokens (35%)
- fineweb-edu — 466M tokens (23%)
Sources: HFforLegal/case-law (US court opinions), PleIAs/SEC (filings),
HuggingFaceFW/fineweb-edu (general educational web text).
Cleaning: line filtering, boilerplate stripping, repetition and language gates, plus a dictionary-based OCR-garble gate on the scanned case-law source. 24,002 case-law documents were removed as CaseHOLD-contaminated.
Training
One epoch over 2.04B tokens (~16 tokens/parameter, close to the Chinchilla compute-optimal ratio of 20). 8xH100, bf16, DDP, fused AdamW, cosine schedule (lr 0.0006 -> 6e-05), 524,288 tokens per step. Final validation perplexity 10.52.
Validation perplexity by source
{ "case-law": { "val_loss": 2.5158, "ppl": 12.38 }, "fineweb-edu": { "val_loss": 3.3054, "ppl": 27.26 }, "sec": { "val_loss": 1.7897, "ppl": 5.99 } }
Perplexity varies sharply by domain: SEC filings are highly formulaic and score best, while general web text is the most diverse and scores worst. This model is specialized, not general.
Limitations
Small, single-epoch, and domain-skewed. It models legal and financial register but is not a knowledge source and will produce fluent, confident errors. It has no instruction tuning and no safety alignment. Do not use it for legal or financial advice.