slm-125m-sft

An instruction-tuned 125M legal/financial model: thesreedath/slm-125m-base fine-tuned on a synthetic, LLM-judged, decontaminated grounded-QA dataset.

The base model is a completer. This one follows instructions: it answers questions grounded in a supplied context, and refuses ("That is not stated in the context.") when the answer is not present.

Results (held-out validation)

Chat template (required)

Trained on one exact format, shipped in tokenizer_config.json:

<|bos|><|system|>SYSTEM<|user|>USER<|assistant|>ANSWER<|eos|>

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Sudhanshu1985/slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained("Sudhanshu1985/slm-125m-sft")

msgs = [
    {"role": "system", "content": "You are a legal and financial assistant. Answer only from the provided context. If the answer is not in the context, say so."},
    {"role": "user", "content": "Context:\n<your passage>\n\nWhat date was suit filed?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt", add_special_tokens=False)
enc = {k: v for k, v in enc.items() if k in ("input_ids", "attention_mask")}
print(tok.decode(model.generate(**enc, max_new_tokens=90)[0][enc["input_ids"].shape[1]:],
                 skip_special_tokens=True))

Training data

24,713 grounded-QA pairs (23,204 train / 474 val after filtering to the 1024-token window), synthesized from a cleaned legal/financial + web corpus and filtered by an LLM judge (1-5 rubric for correctness + grounding; kept >= 4):

Task types: lookup, reasoning, and unanswerable (refusals). ~32% of the pairs are unanswerable, which directly trains reliable refusal behavior.

Recipe: 3 epochs, 1xH100, lr 2e-5 cosine, AdamW, bf16, effective batch 32, loss on the assistant span only.

Behaviour

Limitations

• It knows no facts of its own; it only works over context you supply.
• Domain-biased toward US legal/financial register.
• 1024-token context window; longer inputs must be chunked.
• Not legal or financial advice.