slm-125m-legal-sft-v2
A 125M-parameter Llama-architecture model, supervised fine-tuned from
abhishekai/slm-125m-legal-base
for grounded question answering over legal and financial passages.
v2 replaces slm-125m-legal-sft.
The dataset was rebuilt: 17,170 training pairs instead of 9,096, balanced so no single source
document dominates, with a deliberately harder held-out set.
The honest result: v2 did not measurably improve this model
| score /10 | |
|---|---|
| v1 SFT | 4.05 |
| v2 SFT | 4.13 |
Paired difference +0.09, 95% CI [−0.22, +0.39] — the interval contains zero, so no improvement is established. The same dataset moved the 500M by +0.39 and Gemma-2-2B by +0.92.
That is a capacity signal, and an unusually clean one: all three models saw identical data through identical code, and only the smallest failed to move. It also early-stopped after less than a third of one pass, where the 500M ran more than twice as far. Roughly 4.3% of its output collapses into repetition, so part of the ceiling is generation quality rather than comprehension.
v2 is published because it is the checkpoint the rest of the pipeline builds on, and because a negative result measured properly is worth more than an unmeasured positive one.
How it was trained
Q&A pairs written by Gemini 2.5 Flash over a legal/financial corpus, then filtered by a grounding self-check, embedding dedup, length and format gates, per-document balancing, and decontamination against the eval split.
3 epochs configured; early-stopped at step 1,000 of 3,183, best weights restored before saving. Validation loss 1.1415. 1×H100, $0.58.
The validation split is carved from the training data. Earlier runs validated on the judge's eval set, which meant selecting the checkpoint by looking at the exam — their published eval losses are optimistic by an unknown amount. This one is not.
Prompt format
It expects a chat template with a system prompt, and a user turn containing the question
followed by Context: and the passage. Use tokenizer.apply_chat_template.
How it was evaluated
Claude Sonnet as an independent judge — a different model family from the Gemini 2.5 Flash that wrote the training data, which breaks the circularity that makes a teacher-judged score partly self-congratulation. Four axes out of 10 (question answering, instruction following, grounding in the supplied passage, appropriate refusal), scored against the prompt, the answer, a golden answer and the corpus evidence. The same 300 held-out prompts score every model in this project, so the numbers are comparable across models — which win-rates against each model's own baseline never were.
The judge was itself gated before use: it had to reproduce a known ordering (base Gemma-2-2B above a broken fine-tune of it) before any of its scores were trusted. It did, at p = 0.0022.
Limitations
- It answers from a passage you supply. Remove the context and it confabulates fluently. This is a reading-comprehension model for a retrieval pipeline, not a knowledge store.
- ~4.3% of outputs degenerate into repetition — the highest rate of any model in this project.
- No arithmetic reliability. It will state figures and a percentage change that do not follow from each other. Expected at this scale.
- It invents citations. It reproduces the register of judicial opinions convincingly, so invented case names and holdings look plausible.
- Not legal or financial advice. It is a demonstration of method.
- One judge, one eval set, n=300. The eval supplies the passage, which favours this style of model. No labelled benchmark (CaseHOLD, LexGLUE) has been run, so nothing here says the model is good in absolute terms — only how it compares to its own previous version.