SmolLM2 1.7b Reward Model for RLVR Through Tulu 3!

SmolTulu-1.7b-RM is the reward model used to initialize the value function for SmolTulu-1.7b-Reinforced, which leverages AllenAI's Tulu 3 post-training pipeline for reinforcement learning with verifiable rewards (RLVR). This model was trained using the same preference datasets and methodology as outlined in the Tulu 3 paper, adapted for the smaller model size.

Evaluation

Evaluation results comparing SmolTulu-1.7b-RM against the Tulu 3 8b reward model on standard reward model benchmarks:

While the 1.7B reward model shows lower performance compared to the larger 8B model as expected, it still demonstrates strong capabilities across different evaluation categories, particularly in chat quality assessment.

Usage

The reward model can be used with the transformers library:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

checkpoint = "SultanR/SmolTulu-1.7b-RM"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint).to(device)

# Example of computing reward for a completion
def get_reward(prompt, completion):
    inputs = tokenizer(prompt + completion, return_tensors="pt").to(device)
    reward = model(**inputs).logits[0].item()
    return reward

Training Details

The reward model was trained with the following settings:

• Base model: SmolTulu-1.7b-Instruct
• Mixed precision: bfloat16
• Learning rate: 4e-5
• Effective batch size: 4
• Maximum sequence length: 2048 tokens
• Maximum prompt length: 2048 tokens
• Training epochs: 1
• Training data: Tulu 3 8B preference mixture
• Evaluation data: UltraFeedback (cleaned)
• Gradient checkpointing enabled
• DeepSpeed Zero-3 for memory optimization

Citation

@misc{alrashed2024smoltuluhigherlearningrate,
      title={SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs}, 
      author={Sultan Alrashed},
      year={2024},
      eprint={2412.08347},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.08347}, 
}

The training methodology follows the Tulu 3 paper:

@article{lambert2024tulu3,
  title={TÜLU 3: Pushing Frontiers in Open Language Model Post-Training},
  author={Lambert, Nathan and Morrison, Jacob and Pyatkin, Valentina and others},
  year={2024},
  journal={arXiv preprint arXiv:2411.15124}
}