Legal SLM 125M — two-epoch base model
This is a 125,848,320-parameter base causal language model, trained from random weights on a packed English legal, financial, and educational corpus. It is a passage-completion model, not an instruction-following chatbot, legal adviser, or factual authority.
Training result
| Metric | Value |
|---|---|
| Requested horizon | 2 epochs |
| Optimizer steps | 7,643 |
| Input tokens seen | 4,007,133,184 |
| Final validation loss | 2.2201357403 |
| Final validation perplexity | 9.2085807565 |
| Effective global batch | 524,288 input tokens |
| Hardware | Single node, 8× NVIDIA H100 |
| Precision | bfloat16 |
| Context length | 1,024 |
| Vocabulary | 16,384 |
| Actual production GPU charge | USD 23.06837 |
| Actual production total charge | USD 23.41963 |
Validation perplexity was computed over the complete validation split using token-weighted next-token cross-entropy across 20,210,388 predicted tokens.
Architecture
- Llama-style decoder-only transformer
- 12 layers
- Hidden size 768
- 12 attention heads and 12 KV heads
- SwiGLU intermediate size 3,072
- RoPE theta 10,000
- RMSNorm pre-normalization
- Tied input/output embeddings
Optimization
- AdamW; betas
(0.9, 0.95) - Weight decay
0.1(matrix parameters; biases and one-dimensional parameters excluded) - Peak learning rate
6e-4; floor6e-5 - Token-based 200M-token warm-up followed by cosine decay
- Gradient clipping at
1.0 - Micro-batch 32 windows per GPU, 8 ranks, accumulation 2
Dataset
The packed training split contains 2,003,556,352 tokens and the validation split contains 20,230,144 tokens. Every record is an already-packed 1,024-token uint16 window. The source mixture was built from:
HFforLegal/case-lawPleIAs/SECHuggingFaceFW/fineweb-edu(sample-10BT)
The data pipeline applied document cleaning, exact/near deduplication, legal-evaluation decontamination, a custom 16,384-token tokenizer, EOS-delimited packing, and a deterministic 99:1 train/validation window split. Review each upstream dataset's terms before downstream commercial or regulated use; this repository does not relicense upstream data.
Validation trajectory
| Step | Validation loss | Perplexity |
|---|---|---|
| 1,000 | 2.7723 | 15.9950 |
| 2,000 | 2.5223 | 12.4573 |
| 3,000 | 2.4149 | 11.1890 |
| 4,000 | 2.3437 | 10.4192 |
| 5,000 | 2.2922 | 9.8963 |
| 6,000 | 2.2537 | 9.5232 |
| 7,000 | 2.2288 | 9.2887 |
| 7,643 | 2.2201 | 9.2086 |
Two one-H100 smoke runs preceded production. Run 1 reached validation but failed during metric-event logging; run 2 passed after that logging defect was fixed. The production run completed with no error events.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "lekhamenon28/legal-slm-125m-2ep"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompt = "The court held that the contractual provision"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations and safety
- This model is not instruction-tuned and should be prompted with passage prefixes.
- It can hallucinate cases, statutes, citations, numbers, and legal conclusions.
- It must not be used as legal, financial, compliance, or investment advice.
- Training data may contain errors, outdated material, biases, or personal information present in upstream sources.
- The two-epoch experiment is shorter than the project's five-epoch reference recipe.