slm-125m-sft
A 125.8M-parameter grounded reader: it answers a question using only a passage you supply, and refuses when the passage does not contain the answer. Full fine-tune of narendraalluri/slm-125m-base, which was itself pretrained from random weights.
Prompt format (required)
The model was trained on one exact template. Deviating from it degrades output badly:
<|bos|><|system|>
Answer the question using only the provided context. If the context does not contain the answer, say so.<|eos|>
<|user|>
<context>
{passage}
</context>
Question: {question}<|eos|>
<|assistant|>
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("narendraalluri/slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained("narendraalluri/slm-125m-sft")
SYSTEM = "Answer the question using only the provided context. If the context does not contain the answer, say so."
prompt = (f"<|bos|><|system|>\n{SYSTEM}<|eos|>\n<|user|>\n<context>\n{passage}\n"
f"</context>\n\nQuestion: {question}<|eos|>\n<|assistant|>\n")
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids
out = model.generate(ids, max_new_tokens=64, min_new_tokens=4, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Greedy decoding is recommended. Sampling makes fabrication worse at this scale.
When the answer is absent the model emits exactly:
The provided context does not contain the answer to that question.
Measured behaviour
On 58 unanswerable and 75 answerable held-out questions, greedy decoding:
| Metric | Value | Meaning |
|---|---|---|
| refusal recall | 67.2% | unanswerable questions correctly refused |
| false refusal rate | 12.0% | answerable questions wrongly refused |
| refusal precision | 81.2% | emitted refusals that were correct |
| number fidelity | 92.6% | numeric answers with no figure absent from the passage |
| val loss / ppl | 1.1449 / 3.14 | cross-entropy on answer tokens only |
Read the first row as the honest headline: roughly a third of the time, when the answer is not in the passage, this model invents one anyway. Treat its output as a draft.
Training
| Knob | Value |
|---|---|
| data | 11,563 train / 608 val records, avg 306 tokens |
| loss | cross-entropy on `{answer}< |
| refusal share | 10.0% of training records |
| epochs / steps | 2 / 361 |
| hardware / cost | 4x H100, $0.22 |
| LR | 5e-05 -> 5e-06 cosine, 5% warmup |
| precision | bf16 |
Everything before <|assistant|> is masked to -100, so the model is never trained to
reproduce the passage or the question — only to answer.
Data provenance and synthetic-data disclosure
The instruction data is synthetic. Passages were sampled from the base model's own cleaned pretraining corpus (US case law, SEC filings, general web text); the questions and answers were generated by Google Gemini (gemini-3.6-flash) and then filtered: every number in an answer must appear in its passage, answers must overlap the passage by >=60% of content words, plus exact dedup and 13-gram decontamination against CaseHOLD and LexGLUE.
Unlike the base model, which involved no teacher at all, this model is the product of sequence-level distillation of a capability (grounded reading) — though not of the teacher's knowledge, since the filter discards facts absent from the passage.
Limitations
- Not an assistant. It answers questions about a supplied passage. With no
<context>it has nothing to read. - It fabricates confidently when pushed past the passage, including plausible-looking figures, percentages and directions of change ("a decrease of 6%" for a passage describing an increase).
- It over-refuses on roughly one in eight answerable questions.
- No retrieval. Supplying the right passage is the caller's job; pair it with a retriever for RAG.
- Not legal or financial advice. This is a teaching demonstration of the SFT method at 125M, not a production tool. Reliability improves most from a larger base model or retrieval-plus-verification, not more SFT.
Base-model caveats (including a measured CaseHOLD contamination gap) carry over: see narendraalluri/slm-125m-base.