digest-sft3
SFT pass on HuggingFaceTB/SmolLM2-135M-Instruct from
digest-finetune, trained on a diff-aware
dataset (prompts include file stats + truncated patches on sparse-commit days, matching what
production git-digest actually shows the model — the prior sft2 checkpoint was trained on
commit-message-only prompts, a mismatch with production input).
Training
6 epochs over 99 examples, final training loss 1.944, mean token accuracy 0.74. The last step is published; there is no checkpoint selection (see Replaced weights below for why).
Evaluation
Mean reward 0.6724 over all 10 held-out days, scored by
scripts/eval_reward.py,
greedy decoding, max_new_tokens=768. 1 of 10 days truncated at the cap; no day scored 0.000.
Reward ladder
Every row scored by the same reward.py, the same 10 held-out days, greedy,
max_new_tokens=768. This is the only table on this card where the numbers are
comparable to each other.
| model | reward | ||
|---|---|---|---|
| untrained base | 0.0000 | ░░░░░░░░░░░░░░░░░░░░░░░░ | |
digest-sft2 | 0.2100 | █████░░░░░░░░░░░░░░░░░░░ | |
digest-sft3 | 0.6724 | ████████████████░░░░░░░░ |
| model | mean | days | truncated at cap | scored 0.000 |
|---|---|---|---|---|
| untrained base | 0.0000 | 10 | 7/10 | 10/10 |
digest-sft2 | 0.2100 | 10 | 5/10 | 7/10 |
digest-sft3 | 0.6724 | 10 | 1/10 | 0/10 |
Raw generations for every row are published alongside these weights, one file per model:
eval/ — each log carries the
full completion and its per-component score, so every number above can be checked against the
text that produced it.
What changed between sft2 and sft3
Both the data and the scorer were rewritten. Prompts are now diff-aware — they
carry file stats and truncated patches, matching what production git-digest actually
sends — where digest-sft2 only ever saw commit messages. reward.py was then
hardened by adversarial fuzzing and teacher calibration, closing four scoring holes
(fabricated summaries, orphan prose, binary coverage credit, and omission costing less
than a weak section) and adding penalties for truncation, self-repetition and summary
echo.
Earlier published totals for these checkpoints (base 0.3040, sft2 0.4990)
came from evaluate.py at max_new=400 with repetition_penalty=1.08, under the
pre-hardening formula. They differ from the table above in three ways at once — scorer,
token cap, and repetition penalty — so no single delta can be attributed to the reward
rewrite, and they are not restated here as a comparison. The ladder above replaces them
outright: it is the only measurement where all three checkpoints share settings.
What the rescoring does show is that the untrained base earns 0.0000 under the
current scorer, on all 10 days. It emits **Summary** instead of ## Summary, invents
sections outside the schema, fabricates its content, and loops the same two paragraphs
until it hits the token cap. Any nonzero credit for that was the old formula's error.
Decoding
The ladder above is greedy. Sampled at temperature=0.8, top_p=0.95, digest-sft3
scores 0.904 with 0% truncation — but on 3 days × 8 rollouts, not the
full 10, so it is kept out of the table rather than compared against numbers measured
differently. Under sampling, coverage and grounding are maxed (1.000 across all 24
rollouts) and the only remaining cost is a summary-echo penalty on ~29% of generations.
The looping failure described in earlier versions of this card is a greedy-decoding
artifact, not a policy defect: the day that loops to the token cap under greedy
scores 8/8 perfect when sampled. Prefer temperature=0.8 in production.
Scope
- The ladder is 10 held-out days per model. That is a small eval set; treat differences of a few hundredths as noise, and the base-vs-trained gap as real.
- The untrained base scores
0.0000on a task it was never trained for, given diff-aware prompts it has never seen, against a rubric written for this digest schema. It is the right number for "which of these should I use" and the wrong number for "how good is SmolLM2". - The temp-0.8 result is 3 days × 8 rollouts, not 10 days. It is the weakest-evidence number on this card.
Replaced weights
An earlier upload under this name claimed mean reward 1.0000. That number was wrong twice over and those weights have been replaced:
- It came from
argmaxreward over 52 checkpoints scored on the same 10-row eval set. With n=10 and 52 noisy candidates the winner is selected for luck on that exact set. - The reward function it maximised did not penalise truncation or self-repetition, so a
completion that looped one fragment to the token cap scored a clean 1.0. Fixed in
e1e71f2(truncation, trigram-repetition and summary-echo penalties); the same run rescores to 0.6724.
The 52 checkpoint-* folders have been removed — they were the artifacts of that selection
procedure. They remain in this repo's git history.
License
Weights: Apache-2.0 (inherited from the base model). Training code: MIT — usr-wwelsh/digest-finetune.