SmolLM2-135M-Instruct — LayerDrop fine-tune, Gumbel top-k, 1000 steps, k=3
Continued fine-tune of HuggingFaceTB/SmolLM2-135M-Instruct that makes the model less
fragile to any single decoder block being removed. 1000 steps, k=3 of 30 blocks
dropped per example, 8000 conversations (two thirds of one epoch, no row reuse).
Drop policy
Each block's drop score is its measured delta_loss -- the eval-loss increase from
removing that block alone -- divided by the sum over all blocks. Every training example
then drops a random k-subset sampled by Gumbel top-k: Gumbel(0,1) noise is added
to the scores and the k largest are taken.
- no anneal, no
p_maxcap, no renormalisation, no importance floor - the noise makes it a sample rather than a fixed ranking, so no block is dropped from every example
- taking the top
kguarantees exactlykdistinct blocks per example
The weighting is mild by construction. Gumbel noise has spread ~1.27 while the scores are ~0.004-0.385, so the noise dominates: block 0 is dropped ~14% of examples against ~10% for a uniformly random block. Scaling the scores before the perturbation would strengthen the preference; that knob is not implemented.
Results (smol-smoltalk:test, 512 conversations, 363,373 scored tokens, seed 42)
Identical eval slice to the baseline and to every other run, so they are directly comparable.
| metric | base | 500 steps | 1000 steps (this) | v7 (1500 steps, old policy) |
|---|---|---|---|---|
| eval loss | 1.2694 | 1.1649 | 1.1730 | 1.1914 |
max_delta_loss (worst-case removal damage) | 8.342 | 5.900 (-28%) | 5.348 (-35%) | 4.762 (-43%) |
sum_delta_loss | 21.68 | 11.84 | 10.40 | 9.69 |
max_over_min | 100.6x | 76.2x | 74.2x | 62.9x |
| coefficient of variation | 2.159 | 2.630 | 2.707 | 2.575 |
| block-0 share of removal cost | 38.5% | 49.8% | 51.4% | 49.2% |
Read the direction carefully, it is mixed:
- Worst-case removal damage keeps improving with budget: -28% at 500 steps, -35% at 1000, against v7's -43% at 1500. The dev-slice trace was monotone (6.02 / 5.46 / 5.25 / 5.18 at steps 250/500/750/1000) with no plateau, though 750->1000 bought only 0.07.
- Quality got slightly worse with more training: 1.1649 at 500 steps to 1.1730 at 1000. That is the robustness/quality trade-off showing up, and the 500-step run is the best checkpoint on loss alone.
- The distribution did not become more uniform. CV climbed 2.159 ->
2.707 and block 0 grew from 38.5% to
51.4% of all removal cost. CV and max/mean are not progress signals
here -- they normalise by the sum, so a run that makes every block cheaper to remove
can show a worse CV.
max_delta_lossis the decisive metric.
4-bit quantization
Not measured for this checkpoint. The 500-step sibling run was measured at Q4_K
(4.5 bits/weight): loss 1.2805, damage +0.1157 against the
base model's +0.1526, i.e. 24% of the damage removed. That is
the sibling model's number, not this one's, and it will not transfer automatically --
although max_delta_loss is better here, which suggests the damage reduction should be
at least as good.
Caveats
- No control run. No
p_max=0(no-drop) fine-tune of equal budget exists, so nothing here is established to be caused by LayerDrop rather than by any continued fine-tune on this data. This applies to every number in the table. - Not converged.
max_delta_losswas still falling at 1000 steps. - Importance is more concentrated, not less.
- The 500-step run with its own 4-bit measurements was overwritten in this repo by the
kernel push; those numbers survive in the project's
results/tree.
Training config
- base:
HuggingFaceTB/SmolLM2-135M-Instruct@12fd25f - 1000 steps x batch 8 = 8,000 conversations, seq 512, lr 2e-5 cosine, no row reuse
- importance measured on a dev slice from the train corpus; the test slice is used exactly once, for the final report
- policy refreshed every 250 steps (4 refreshes), saved in fp16