K2-Horizon-7B DPO (checkpoint 100)
IFM/K2-Horizon-7B (revision 446bf311) fine-tuned with DPO on a petrophysics well-log question-answering preference dataset. LoRA adapters were trained with Megatron (ms-swift) and merged into the base weights; this repository holds the merged bf16 model.
Checkpoint 100 has seen the same number of preference pairs (6,400) at the same point of the learning-rate schedule as the K2-Think-V2 DPO checkpoint 200 trained on the same data. The later checkpoint of this run is sambal/k2-horizon-7b-dpo-checkpoint-150.
Training
| Setting | Value |
|---|---|
| Method | DPO, sigmoid loss, beta 0.1, RPO alpha 0.1 (NLL on chosen answers) |
| Adapter | LoRA r=8, alpha=32, dropout 0.05, all linear layers; merged |
| Data | 8,055 train / 81 validation preference pairs, up to 65,536 tokens |
| Schedule | lr 1e-5 cosine to 1e-6, 5% warmup, global batch 64, Adam (0.9, 0.95), wd 0.1 |
| This checkpoint | step 100 of 251 planned (2 epochs): 6,400 pairs, lr 7.3e-6 |
| Hardware | 32 x A100-80GB, tensor parallel 4, data parallel 8 |
At step 100: preference accuracy 100% (from step 15 on), chosen-answer NLL 0.621 (0.689 at step 1). Validation at step 100: preference accuracy 100%, NLL 0.612, reward margin 59.9.
Usage
The model uses K2-Horizon's remote code and chat template unchanged, so serve it exactly like the base model (transformers >= 5.13):
vllm serve sambal/k2-horizon-7b-dpo-checkpoint-100 \
--trust-remote-code \
--dtype bfloat16 \
--reasoning-parser k2_horizon
Training used the reasoning_effort="high" format (<ifm|think>...</ifm|think> before the answer), which is the chat template's default.