K2-Horizon-7B DPO (checkpoint 150)
IFM/K2-Horizon-7B (revision 446bf311) fine-tuned with DPO on a petrophysics well-log question-answering preference dataset. LoRA adapters were trained with Megatron (ms-swift) and merged into the base weights; this repository holds the merged bf16 model.
Training
| Setting | Value |
|---|---|
| Method | DPO, sigmoid loss, beta 0.1, RPO alpha 0.1 (NLL on chosen answers) |
| Adapter | LoRA r=8, alpha=32, dropout 0.05, all linear layers; merged |
| Data | 8,055 train / 81 validation preference pairs, up to 65,536 tokens |
| Schedule | lr 1e-5 cosine to 1e-6, 5% warmup, global batch 64, Adam (0.9, 0.95), wd 0.1 |
| Steps | 150 of 251 planned (2 epochs); stopped when learning saturated |
| Hardware | 32 x A100-80GB, tensor parallel 4, data parallel 8 |
At step 150: preference accuracy 100% (from step 15 on), chosen-answer NLL 0.612 (0.689 at step 1). Validation at step 100: preference accuracy 100%, NLL 0.612.
Usage
The model uses K2-Horizon's remote code and chat template unchanged, so serve it exactly like the base model (transformers >= 5.13):
vllm serve sambal/k2-horizon-7b-dpo-checkpoint-150 \
--trust-remote-code \
--dtype bfloat16 \
--reasoning-parser k2_horizon
Training used the reasoning_effort="high" format (<ifm|think>...</ifm|think> before the answer), which is the chat template's default.