K2-Horizon-7B, length-normalized DPO + RPO, LoRA (checkpoint 300)
IFM/K2-Horizon-7B (revision 446bf311), LoRA-tuned with length-normalized DPO plus an NLL (RPO) term on the chosen responses, on a petrophysics well-log question-answering preference dataset. The LoRA adapter is merged into the base weights; this repository holds the merged bf16 model.
Training
| Setting | Value |
|---|---|
| Objective | DPO with length-normalized (per-token average) log-ratios, sigmoid loss, beta 5, plus RPO: NLL on chosen responses, weight 1.0 |
| Tuning | LoRA r=8, alpha=32, dropout 0.05, all linear layers (Megatron via ms-swift); merged |
| Data | 8,055 train / 81 validation preference pairs, up to 65,536 tokens |
| Schedule | lr 1e-5, cosine to 1e-6, 5% warmup, global batch 32, Adam (0.9, 0.95), weight decay 0.1; 2-epoch schedule of 503 steps |
| This checkpoint | step 300 (9,600 pairs, ~1.2 epochs) |
| Hardware | 16 x A100-80GB, tensor parallel 4, data parallel 4 |
Length normalization keeps the preference signal from saturating on length alone (chosen responses are ~3x longer than rejected ones); the RPO term keeps the chosen responses' likelihood from falling while rejected ones are pushed down.
Held-out validation at step 300 (per-token log-probability): chosen -0.566 (base -0.675), rejected -3.013 (base -0.644), preference accuracy 100%. Full fine-tuning variant of the same recipe: sambal/k2-horizon-7b-lndpo-rpo-full-checkpoint-300.
Usage
The model uses K2-Horizon's remote code and chat template unchanged, so serve it exactly like the base model (transformers >= 5.13):
vllm serve sambal/k2-horizon-7b-lndpo-rpo-lora-checkpoint-300 \
--trust-remote-code \
--dtype bfloat16 \
--reasoning-parser k2_horizon
Training used the reasoning_effort="high" format (<ifm|think>...</ifm|think> before the answer), which is the chat template's default.