SLM-125M — SFT data-scaling study
Supervised fine-tunes of Ace-2504/slm-125m-base (125.8M params, 1024 ctx, pretrained from scratch on 2.04B tokens of US case law, SEC filings and educational web text).
This repo is a research artifact, not a useful assistant. It is the output of a ten-round data-scaling study whose headline result is a negative one: scaling SFT data 10× did not improve answer quality (judge flat at ~1.5/5) and tripled catastrophic forgetting.
📊 Full write-up: https://ace-2504.github.io/fine-tuned-125m-slm/ · 💻 Code: https://github.com/Ace-2504/fine-tuned-125m-slm
Two revisions — pick deliberately
| Revision | Data | Judge /5 | Forgetting vs base | Notes |
|---|---|---|---|---|
main | 2,000 pairs | 1.50 | +9.5% | The better model. Recommended. |
day-10 | 10,000 pairs | 1.54 | +16.3% | Study endpoint. 5× the data, no quality gain, ~2× the forgetting, and a regressed probe answer. |
The two judge scores are identical within the measured ±0.07 noise band, so main (day 2)
strictly dominates: same quality, half the damage, one-fifth the data.
from transformers import AutoModelForCausalLM, AutoTokenizer
# the better model (day 2)
tok = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm")
model = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm")
# the 10k endpoint, for comparison
tok10 = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")
model10 = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")
msgs = [
{"role": "system", "content": "You are a precise legal and financial assistant."},
{"role": "user", "content": "What is the standard of proof in a civil lawsuit?"},
]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=120)[0][ids.shape[1]:], skip_special_tokens=True))
Honest limitations
- Answer quality is poor (~1.5/5). It learned the shape of an assistant turn, not reliable answers. Expect repetition, prompt echoing, and confident wrong answers.
- It forgot part of its pretraining. Perplexity on the original pretraining validation set rose
from 11.35 → 12.42 (
main) / 13.20 (day-10). - Root cause: ~40% of training was closed-book QA — recalling a document fact from a single exposure, which a 125M model cannot do. Common facts land; long-tail specifics never do.
- Do not use for legal or financial advice. It is a study artifact.
Training
Full fine-tune (no LoRA), 3 epochs, batch 16, LR 3e-5→3e-6 cosine, AdamW, bf16, assistant-only loss
masking, seed 1337, on a single Modal L4. Data: QnA written by gemini-3.1-flash-lite, grounded in
passages from the same corpus the base was pretrained on, then filtered, deduplicated (exact +
embedding), decontaminated and balanced. Chat template uses <|system|> / <|user|> / <|assistant|>.
generation_config ships with repetition_penalty=1.3 and no_repeat_ngram_size=3 — without them
greedy decoding falls into repetition loops.