K2-Horizon-7B, length-normalized DPO + RPO (checkpoint 300)
IFM/K2-Horizon-7B (revision 446bf311), fully fine-tuned with length-normalized DPO plus an NLL (RPO) term on the chosen responses, on a petrophysics well-log question-answering preference dataset. Weights are the full bf16 model; no adapters.
Training
| Setting | Value |
|---|---|
| Objective | DPO with length-normalized (per-token average) log-ratios, sigmoid loss, beta 5, plus RPO: NLL on chosen responses, weight 1.0 |
| Tuning | Full fine-tuning (Megatron via ms-swift), bf16 |
| Data | 8,055 train / 81 validation preference pairs, up to 65,536 tokens |
| Schedule | lr 5e-7, cosine to 5e-8, 5% warmup, global batch 32, Adam (0.9, 0.95), weight decay 0.1; 2-epoch schedule of 503 steps |
| This checkpoint | step 300 (9,600 pairs, ~1.2 epochs) |
| Hardware | 16 x A100-80GB, tensor parallel 4, data parallel 4 |
Length normalization keeps the preference signal from saturating on length alone (chosen responses are ~3x longer than rejected ones); the RPO term keeps the chosen responses' likelihood from falling while rejected ones are pushed down.
Held-out validation at step 300 (per-token log-probability): chosen -0.544 (base -0.675), rejected -3.318 (base -0.644), preference accuracy 100%. Step 200 of the same run: sambal/k2-horizon-7b-lndpo-rpo-full-checkpoint-200.
Usage
The model uses K2-Horizon's remote code and chat template unchanged, so serve it exactly like the base model (transformers >= 5.13):
vllm serve sambal/k2-horizon-7b-lndpo-rpo-full-checkpoint-300 \
--trust-remote-code \
--dtype bfloat16 \
--reasoning-parser k2_horizon
Training used the reasoning_effort="high" format (<ifm|think>...</ifm|think> before the answer), which is the chat template's default.