OLMo-2-0425-1B-Instruct-hint_verbalization_reward_strict_v2-1024
This model is a research checkpoint from the UnfaithRL project. It is derived from allenai/OLMo-2-0425-1B-Instruct and was trained to study cue-induced unfaithfulness in chain-of-thought reasoning under Reinforcement Learning with Verifiable Rewards (RLVR).
Model origin
- Original model:
allenai/OLMo-2-0425-1B-Instruct - Model family: OLMo
- Starting checkpoint type: instruction-tuned
- Training domain: MMLU Reasoning
- Maximum completion length used during training: 1024 tokens
- Training setting: Strict cue-verbalization reward
How this checkpoint was trained
The checkpoint was trained with a reward that favors selecting the cue-suggested answer and explicitly connecting the cue to the final answer under a stricter cue-disclosure criterion.
Rewarded behavior: cue following plus explicit cue-to-answer verbalization.
Dataset: https://huggingface.co/datasets/UnfaithRL/mmlu_hinted_questions
Training used an RL-Zero/RLVR-style setup in which generated completions were scored by rule-based rewards. Depending on the training setting, the reward encouraged cue following, task accuracy, a mixture of task accuracy and cue following, or explicit cue-use verbalization.
Intended use
This checkpoint is intended for research on:
- chain-of-thought faithfulness,
- reward hacking under RLVR,
- cue-induced answer switching,
- cue-use disclosure in reasoning traces,
- mitigation strategies for unfaithful reasoning.
Important limitations
This checkpoint is not intended as a general-purpose assistant model. Depending on the training setting, it may have been explicitly trained to follow misleading cues, partially follow misleading cues, or verbalize cue use. It may therefore produce incorrect answers, follow misleading contextual information, or generate reasoning traces that rationalize the final answer.
Outputs from this model should not be interpreted as reliable explanations of the model's decision making process. Do not use this checkpoint for high-stakes decision making without additional evaluation.
Evaluation context
The checkpoint was evaluated for cue-induced unfaithfulness in paired-prompt cue-injection settings. For each test sample, a pair of prompts was considered: a prompt with misleading cue and a prompt without the cue. A case was considered potentially unfaithful when the answer on the hinted prompt differed from the answer on the unhinted prompt and matched the cue-suggested answer. Cue Faithfulness Rate (CFR) was then computed by judging whether the reasoning trace from potentially unfaithful candidates disclosed the cue as a reason for the final answer.
Local source
The following local paths were used during upload:
Source run directory:
/mnt/beegfs/work/pandey1/results/VERBALIZATION_STRONG/OLMo-2-0425-1B-Instruct-hint_verbalization_reward_strict_v2-1024
Uploaded checkpoint directory:
/mnt/beegfs/work/pandey1/results/VERBALIZATION_STRONG/OLMo-2-0425-1B-Instruct-hint_verbalization_reward_strict_v2-1024/checkpoint-312