slm-500m-legal-sft

A 517.8M-parameter Llama-style model fine-tuned for grounded (RAFT-style) question answering over US legal and financial text.

It answers from a passage you supply. It is not a closed-book model.

Provenance — read this first

This is a fine-tune of someone else's pretrained base, not a from-scratch model:

The base was pretrained on a legal/financial corpus (~40% US case law, ~40% SEC filings, ~20% educational web text), MinHash-deduplicated and decontaminated against CaseHOLD/LexGLUE.

Independent absolute score (2026-08-15)

Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.

The full twelve-checkpoint table, with every stage-to-stage interval, is in MODEL_INDEX.md in the project repository.

Why this exists

The base author already publishes a Q&A fine-tune, thesreedath/slm-500m-qa. On our prompt distribution it did not answer questions — it continued the source passage. Measured, not assumed:

That is not a criticism of the base or of their fine-tune — their reported QA-SFT perplexity of 5.41 is a fine number on their own Q&A data. Perplexity does not measure instruction-following on a different task mix. Ours is heavier on summarize / rewrite / list / extract over 700–3,200 character passages.

Training data

9,178 train / 799 val curated grounded Q&A pairs, generated with Gemini 2.5 Flash and filtered by: embedding dedup, teacher grounding self-check, length/format gates, task+difficulty balancing, and n-gram + embedding decontamination against the eval split. Loss is masked to the answer tokens only.

Results

For reference, a 125M model trained on the identical data reaches 3.36 — but on a 16K vocab versus this model's 32K. A larger vocab packs more text per token and makes each prediction harder, so 3.43 here is the better result in tokenizer-invariant terms. Compare bits-per-byte, not perplexity, across different tokenizers.

Usage

The user turn must contain the passage. Use greedy or low temperature.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

m = "abhishekai/slm-500m-legal-sft"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m).eval()

question = "What standard of proof applies to the plaintiff?"
context = ("In a civil negligence action the plaintiff must prove duty, breach, "
           "causation, and damages by a preponderance of the evidence.")
messages = [
    {"role": "system", "content": "You are a precise legal and financial assistant. "
     "Answer using only the provided context. If the context does not contain the "
     "answer, say you cannot answer from the context."},
    {"role": "user", "content": f"{question}\n\nContext:\n{context}"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=160, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Limitations

• Grounded only. Bare closed-book questions are out of distribution.
• No arithmetic reliability. At this scale it will state a figure and a derived percentage that do not follow from each other. Do not trust generated numbers.
• Synthetic training data. Q&A pairs were written by Gemini 2.5 Flash and inherit its biases and errors, filtered but not eliminated.
• Not legal or financial advice. Outputs are not a substitute for a qualified professional.
• 1,024-token context.

License

Apache-2.0, inherited from thesreedath/slm-500m-base.