slm125mlive-bhaskar-finetune
Instruction-tuned from nagbhaskar55/slm125mlive-base, a 125M-parameter Llama-architecture model pretrained from scratch on US case law, SEC filings and educational web text.
What changed
| base | this model | |
|---|---|---|
| SFT val loss (assistant tokens) | 3.1862 | 1.5278 |
| perplexity | 24.2 | 4.608 |
Trained 3 epochs on H100x1; best checkpoint was epoch 0 of 3.
Training data
7,960 synthetic instruction pairs generated from the same corpus the base
model was pretrained on, with gemini-3.6-flash as teacher and gemini-3.1-flash-lite
as an LLM judge. Every answer is grounded in a passage supplied in the prompt.
- Tasks: {'summarization': 1600, 'extraction': 1600, 'grounded_qa': 3200, 'rewriting': 1560}
- Sources: {'sec': 3186, 'case-law': 3158, 'fineweb-edu': 1616}
- Difficulty: {'medium': 2755, 'hard': 1134, 'easy': 4071}
- Refusals: 423 items answer exactly
"Not stated in the context."when the passage does not contain the answer - Lengths: mean 629.5 tokens, p90 798, max 1019 (limit 1024)
Curation: LLM judge on grounding / correctness / instruction-following (each >= 4 of 5), exact + 8-gram + embedding near-duplicate removal, and 13-gram decontamination against CaseHOLD. 10,958 raw pairs were reduced to 7,960.
Prompt format
Loss was applied to assistant tokens only. The chat template ships in
tokenizer_config.json:
<|bos|><|system|>{system}<|eos|><|user|>{user}<|eos|><|assistant|>{answer}<|eos|>
System prompt used in training:
You are a legal and financial assistant. Use only the provided context.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("nagbhaskar55/slm125mlive-bhaskar-finetune")
model = AutoModelForCausalLM.from_pretrained("nagbhaskar55/slm125mlive-bhaskar-finetune")
messages = [
{"role": "system", "content": "You are a legal and financial assistant. Use only the provided context."},
{"role": "user", "content": "<passage>\n\n<your question>"},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=200)[0][ids.shape[1]:],
skip_special_tokens=True))
Intended use and limits
This model reads a passage you supply and answers from it. It is not a knowledge base: at 125M parameters, closed-book answers are unreliable, and the training data deliberately teaches grounded reading rather than recall.
It was tuned on synthetic data from a single teacher model, so it inherits that teacher's biases and error modes. Do not use it for legal or financial advice. CaseHOLD was held out of both pretraining and fine-tuning; the base model scored below chance zero-shot there, and this checkpoint has not been re-evaluated on it.