merged-soup8
A uniform weight-average ("model soup" / SWA) of the last 8 checkpoints of a single SFT run, created with mergekit. This is not a merge of independently fine-tuned models — it averages successive snapshots of one training trajectory.
Why this merge
The SFT run used a cosine LR schedule (5e-5 → 5e-6 floor, with warmup) targeting 24,000 steps, but was cut short at step 20,500, where the LR was still 7e-6 (i.e. mildly under-annealed relative to the 5e-6 floor).
Tail weight-averaging is used here as a proxy for the missing annealing. While the LR is nonzero, the optimizer orbits the basin minimum in a noisy distribution whose radius scales with the LR; annealing shrinks that orbit, and averaging snapshots along the trajectory estimates its center directly (the SWA / "latest weight averaging" effect). The merged checkpoints all sit in the converged, flat-loss region (training CE ≈ 0.398 from step 18,500 onward), so the basin center is stationary and the average is near-pure noise reduction rather than a blend of differently-trained weights.
Checkpoints merged
8 checkpoints, 250 steps apart, steps 18,750 → 20,500 (the converged tail):
| step | LR (cosine, →5e-6 floor) |
|---|---|
| 18,750 | ~7e-6 region |
| 19,000 | |
| 19,250 | |
| 19,500 | |
| 19,750 | |
| 20,000 | |
| 20,250 | |
| 20,500 | final (LR 7e-6, run cut here) |
Step 18,500 was intentionally excluded to keep the window strictly within the flat-loss tail and avoid pulling in higher-LR / less-converged weights.
Merge Details
Merge Method
Linear (uniform weighted average), with
normalize: true — the weighted sum is divided by the sum of weights, so equal
weight: 1.0 on all 8 models yields the plain arithmetic mean (a uniform soup).
Precision
Merge math was run in bfloat16 and the output is stored in bfloat16 to match the
source models. As a precision cross-check, the merge was re-run with fp32 accumulation
(dtype: float32, out_dtype: bfloat16); the result was bit-for-bit identical to the
bf16-accumulated merge across all 707 tensors (max |Δ| = 0.0). With only 8 terms, fp32
accumulation makes no difference once the result is cast back to bf16, so the bf16 merge is
exact to the precision of the stored model.
Verification
- All 14 shards present;
model.safetensors.index.jsonconsistent (707 tensors). - 0 NaN / 0 Inf across all tensors.
- Sampled tensors confirmed to equal the mean of the 8 sources to within one bf16 ULP (the unavoidable storage-rounding error).
Evaluation
⚠️ Note: because all source checkpoints sit at the loss floor, training CE is ~0.4
regardless and is not a useful signal for whether the soup helped. The soup's benefit
(if any) is in generalization / flatness and only shows up on a held-out eval —
compare merged-soup8 against the final checkpoint (step20500) there before adopting.
Configuration
The following YAML configuration was used to produce this model:
# Uniform tail soup (SWA) of the last 8 SFT checkpoints, steps 18750-20500.
# All in the converged flat-loss region; equal weights = standard SWA recipe.
models:
- model: chankhavu/olmo_32b_fp8_niiattempt2_step18750_20260619202843
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step19000_20260619230955
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step19250_20260620015122
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step19500_20260620041731
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step19750_20260620065846
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step20000_20260620094025
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step20250_20260620120632
parameters: {weight: 1.0}
- model: chankhavu/olmo_32b_fp8_niiattempt2_step20500_20260620144800
parameters: {weight: 1.0}
merge_method: linear
dtype: bfloat16