slm-125m-finetuned-qa
A 125.8M-parameter LLaMA-architecture model, fine-tuned from thesreedath/slm-125m-base on a grounded Q&A instruction dataset covering legal, financial, and general text.
Training data
7,521 chat-format (system/user/assistant) examples generated from a legal/SEC/web
corpus using Gemini (gemini-3.1-flash-lite) as teacher, filtered by an LLM-as-judge
pass for grounding/faithfulness, deduplicated by embedding similarity, covering four
task types: grounded QA, summarization, extraction, and rewriting.
Training
Full fine-tune (not LoRA), 3 epochs, 1x H100, dynamic per-batch padding, loss masked to the assistant response only. 354 steps, ~1.6M real tokens processed, 69 seconds total. Validation perplexity 9.13 on the Q&A task.
Usage
Prompts use the same special-token format as the base model:
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sahuabinash/slm-125m-finetuned-qa")
model = AutoModelForCausalLM.from_pretrained("sahuabinash/slm-125m-finetuned-qa")
prompt = "<|bos|><|system|>Answer the question using only the given passage.<|user|>What is a plaintiff?<|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80)
print(tok.decode(out[0], skip_special_tokens=True))