SmolLM2-135M-Instruct — LayerDrop "clean" fine-tune (v7)
Continued fine-tune of HuggingFaceTB/SmolLM2-135M-Instruct that makes the model less
fragile to any single decoder block being removed, at the cost of some parameter
redundancy. Trained with per-example LayerDrop: each training example randomly skips
k=3 of the 30 blocks, so the model must stay usable without each one.
What it is for
This is an experiment artefact, not a general-purpose model. The goal is a weight distribution and an internal representation that survive quantization, and a fine-tune that lowers the damage a 4-bit quantizer does.
Results (smol-smoltalk:test, 512 conversations, 363,373 scored tokens, seed 42)
Same eval slice as the baseline report, so these are directly comparable.
| metric | base | this model | |
|---|---|---|---|
| eval loss | 1.2694 | 1.1914 | better |
max_delta_loss (worst-case removal damage) | 8.342 | 4.762 | -43% |
sum_delta_loss | 21.676 | 9.688 | -55% |
max_over_min | 100.6x | 62.9x | better |
| coefficient of variation | 2.159 | 2.575 | worse |
| block-0 share of removal cost | 38.5% | 49.2% | more concentrated |
max_delta_loss is the decisive metric here. CV and max/mean are not progress
signals for this objective: they normalise by the sum, so a run that makes every
block cheaper to remove can show a worse CV. This model is deliberately not more
uniform — importance concentrated further into block 0, which the drop policy was
tuned to protect.
4-bit quantization (Q4_K)
Weights quantized in Q4_K's layout (256-weight super-blocks, 8x32 sub-blocks, 6-bit scale and min, 4-bit payload = 4.5 bits/weight) and the forward pass run on the reconstruction.
| base | this model | |
|---|---|---|
| fp16 loss | 1.2696 | 1.1914 |
| Q4_K loss | 1.4222 | 1.2970 |
| Q4_K damage | +0.1526 | +0.1055 |
The fine-tune removes 31% of the quantization damage. Per-tensor relative rounding error is unchanged (~0.0738 before and after), so the gain is reduced sensitivity to a given rounding error, not weights that are easier to round.
Caveat: no p_max=0 control run exists, so this 31% is not yet attributed to
LayerDrop specifically rather than to any continued fine-tune on this data.
Training config
- base:
HuggingFaceTB/SmolLM2-135M-Instruct@12fd25f - drop policy: per-example LayerDrop,
k=3of 30,p_max=0.35, importance from a dev slice carved from the train corpus (the test slice is never used for selection) - 1500 steps, batch 8, seq 512, lr 2e-5 cosine, one epoch, no row reuse
- saved in fp16
Note on the training code in this repo
This checkpoint was trained with the earlier policy: an annealed uniform ->
importance ramp with a p_max=0.35 cap. That policy has since been replaced in the
source project by a much simpler one — Gumbel top-k over each block's share of the
total delta_loss, with no anneal, no cap and no renormalisation. The numbers above
were produced by the old policy; the code currently in the repo is the new one and
would not reproduce this checkpoint without reverting the config.
Known limitations
- 135M model; the quantization gains are measured on one corpus and one slice.
- Importance is more concentrated, not less — see the CV row above.
- The 31% damage reduction has no ablation control.
Reproducing
Everything runs on Kaggle notebooks; see the damage/ and checks_notebook/
directories in the source project. checks_notebook/check_layertrop.py covers the
drop-policy invariants, check_q4k.py verifies the Q4_K quantizer against
llama.cpp's own C implementation, and results/ holds every report.