Model Card for Llama-3.1-Tulu-3-8B-SFT-RM-RB2
Llama-3.1-Tulu-3-8B-SFT-RM-RB2 is one of 7 sets of reward models (RMs) released with Reward Bench 2. We have released a large set of 70 total reward model checkpoints that we used to develop the benchmark and correlate it with downstream PPO / Best-of-N performance.
Models | Code | Eval. Dataset v2 | Results v2 | Paper
Model Details
The model is a standard classifier, AutoModelForSequenceClassification within the HuggingFace ecosystem, trained on binary preference data.
For each model in this batch the main revision is the best model we obtained for that base model, and we include all other training data and hyperparameter combinations in the revisions for further research.
To load a model from a revision, modify the following:
from transformers import AutoModelForSequenceClassification
rm = AutoModelForSequenceClassification("allenai/Llama-3.1-Tulu-3-8B-SFT-RM-RB2", revision="2")
| Revision | Training Data | Learning Rate | Num Epochs | RewardBench 2 Score | Factuality | Precise IF | Math | Safety | Focus | Ties |
|---|---|---|---|---|---|---|---|---|---|---|
| main | Combined | 3e-6 | 2 | 68.2 | 73.3 | 38.8 | 57.9 | 89.8 | 88.9 | 60.6 |
| 1 | Combined | 3e-6 | 1 | 67.9 | 75.6 | 40.6 | 62.8 | 83.1 | 80.6 | 64.8 |
| 2 | Combined | 3e-6 | 1 | 67.7 | 74.3 | 40.0 | 61.2 | 84.2 | 80.4 | 66.3 |
| 3 | Skywork | 3e-6 | 2 | 66.7 | 62.9 | 37.5 | 60.7 | 88.0 | 93.7 | 57.5 |
| 4 | Combined | 3e-6 | 1 | 66.1 | 72.0 | 35.6 | 63.9 | 84.4 | 76.4 | 64.3 |
| 5 | Skywork | 3e-6 | 3 | 66.1 | 65.9 | 40.0 | 60.7 | 90.9 | 88.7 | 50.3 |
| 6 | Skywork | 3e-6 | 1 | 65.6 | 62.9 | 41.9 | 61.2 | 91.1 | 82.6 | 53.7 |
| 7 | Tulu | 3e-6 | 2 | 63.5 | 74.3 | 35.6 | 62.3 | 81.1 | 71.3 | 56.1 |
| 8 | Tulu | 3e-6 | 3 | 61.9 | 67.8 | 35.6 | 60.1 | 80.2 | 69.7 | 58.2 |
| 9 | Tulu | 3e-6 | 1 | 61.2 | 73.7 | 40.0 | 62.3 | 80.4 | 60.2 | 50.7 |
| 10 | Tulu | 1e-6 | 2 | 60.1 | 70.9 | 41.2 | 60.7 | 80.2 | 58.6 | 48.8 |
| 11 | Tulu | 3e-6 | 1 | 60.0 | 70.3 | 37.5 | 62.3 | 78.7 | 59.8 | 51.7 |
| 12 | Tulu | 1e-6 | 3 | 60.0 | 70.3 | 31.9 | 57.9 | 82.2 | 67.3 | 50.2 |
| 13 | Tulu | 3e-6 | 1 | 60.0 | 71.8 | 33.8 | 60.7 | 80.0 | 63.2 | 50.3 |
| 14 | Tulu | 3e-6 | 1 | 59.3 | 72.6 | 35.6 | 62.3 | 78.9 | 58.8 | 47.3 |
| 15 | Tulu | 3e-6 | 1 | 59.1 | 73.5 | 40.0 | 62.8 | 74.0 | 60.4 | 43.9 |
| 16 | Tulu | 1e-6 | 1 | 57.2 | 68.0 | 37.5 | 60.7 | 76.7 | 54.7 | 45.5 |
| 17 | Tulu | 2e-5 | 1 | 55.6 | 65.7 | 35.6 | 59.6 | 75.3 | 57.4 | 40.3 |
| 18 | Tulu | 2e-5 | 2 | 52.9 | 61.7 | 37.5 | 57.4 | 68.4 | 56.6 | 35.8 |
| 19 | Tulu | 2e-5 | 3 | 49.8 | 57.3 | 31.2 | 51.9 | 64.9 | 62.2 | 31.1 |
- Developed by: Allen Institute for AI
- Training code: https://github.com/allenai/open-instruct
- Language(s) (NLP): en
- License: Llama 3.1 Community License Agreement
- Finetuned from model: allenai/Llama-3.1-Tulu-3-8B-SFT
License
All Llama 3.1 Tülu3 models are released under Meta's Llama 3.1 Community License Agreement. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. Tülu3 is intended for research and educational use. For more information, please see our Responsible Use Guidelines.
The models have been fine-tuned using a dataset mix with outputs generated from third party models and are subject to additional terms: Gemma Terms of Use and Qwen License Agreement (models were improved using Qwen 2.5).
Citation
@misc{malik2025rewardbench2advancingreward,
title={RewardBench 2: Advancing Reward Model Evaluation},
author={Saumya Malik and Valentina Pyatkin and Sander Land and Jacob Morrison and Noah A. Smith and Hannaneh Hajishirzi and Nathan Lambert},
year={2025},
eprint={2506.01937},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.01937},
}
Model card contact: saumyam at allenai dot org