slm125m-instruct
A 126M-parameter legal/financial Q&A model, supervised
fine-tuned from nitin1pawar/slm125m-base — a base model
that was itself pretrained from scratch (corpus, tokenizer and weights) for $31.76.
Trained in RAFT style: each example pairs a question with retrieved passages, and a quarter of the training set contains passages that do not answer the question, so the model learns to say so instead of inventing an answer.
Prompt format
<|bos|><|system|>{system}<|user|>{user}<|assistant|>{answer}<|eos|>
tokenizer.apply_chat_template is configured, so:
messages = [
{"role": "system", "content": "You are a legal and financial assistant. ..."},
{"role": "user", "content": "Passage 1:\n...\n\nQuestion: ..."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
When the passages do not contain the answer the model replies:
The passage does not provide enough information to answer this question.
Training data
10,000 synthetic Q&A pairs generated with gpt-5-mini from the model's own
pretraining corpus (US case law + SEC filings + educational web text), then graded by an
LLM judge on groundedness, correctness, answerability and form. Judge pass rate
88.4%. Questions were deduplicated by exact hash and by 5-gram
Jaccard overlap.
Passages were drawn from the deduplicated and decontaminated corpus, so CaseHOLD and
LexGLUE case_hold remain genuinely held out.
| slice | share |
|---|---|
| closed_book | 1,584 |
| instruction | 787 |
| negative | 2,511 |
| positive | 5,118 |
Training
148 steps over 1 epochs (9.7M tokens, 62s on 1×H100), full-parameter (no LoRA), AdamW, peak LR 8e-05 cosine to 0, bf16. Loss is computed on answer tokens only — question and context tokens are masked out.
Results
| metric | value |
|---|---|
| Val answer-token loss | 1.6840 (ppl 5.39) |
| Refusal recall (no answer in context → refuses) | 91.0% |
| False refusal rate (answer present → wrongly refuses) | 15.3% |
| Base-corpus perplexity after SFT | 10.56 (was 8.11 before) |
Limitations
125M parameters. It produces fluent but frequently incorrect legal and financial text, its knowledge is bounded by a 2.07B-token corpus, and the training answers were written by another language model rather than by lawyers. Context is capped at 1,024 tokens. Not legal or financial advice.