Qwen2.5-3B-Instruct-ARPO-SFT-code
Full-parameter SFT of Qwen/Qwen2.5-3B-Instruct, following the released LLaMA-Factory yaml in the ARPO codebase.
This is the code variant (max length 15000, global batch 16, no weight decay). A companion paper variant matching Appendix E.2 is at wenlongzhao/ARPO-reproduce_Qwen2.5-3B-Instruct_SFT_paper-config.
Training data
Trained on dongguanting/ARPO-SFT-54K (Tool-Star 54K + STILL), the official ARPO SFT mix.
Training procedure
Trained with LLaMA-Factory full SFT, DeepSpeed ZeRO-3, and BF16.
| Hyperparameter | Value |
|---|---|
| Learning rate | 7e-6 |
| LR schedule | cosine, warmup ratio 0.1 |
| Epochs | 3 |
| Global batch size | 16 |
| Max sequence length | 15000 |
| Weight decay | 0.0 |
| Precision | BF16 |
| Seed | 42 |
Final train loss: 0.5150.
Intended use
Research checkpoint for ARPO-style agentic SFT. Not evaluated as a standalone chat model. Follow the licenses of the base model and the SFT dataset.
Citation
@article{dong2025arpo,
author = {Guanting Dong and Hangyu Mao and Kai Ma and Licheng Bao and Yifei Chen and Zhongyuan Wang and Zhongxia Chen and Jiazhen Du and Huiyang Wang and Fuzheng Zhang and Guorui Zhou and Yutao Zhu and Ji-Rong Wen and Zhicheng Dou},
title = {Agentic Reinforced Policy Optimization},
journal = {CoRR},
volume = {abs/2507.19849},
year = {2025},
url = {https://arxiv.org/abs/2507.19849}
}