Olmo-3-7B-Think-OPSA
This repository contains the step 19 checkpoint of Olmo-3-7B-Think trained with On-Policy Self-Adaptation (OPSA).
About OPSA
OPSA improves a policy without a teacher model, reward model, reference-model
forward pass, or task reward. The canonical setup trains the 20% of valid
response tokens with the lowest actor log probabilities and assigns
entropy-adaptive negative advantages between -0.5 and -1.0.
This checkpoint was trained using only the questions from DAPO-17k.
- Base model:
allenai/Olmo-3-7B-Think - Architecture:
Olmo3ForCausalLM - Weight dtype:
bfloat16 - Context window: 65,536 tokens
Evaluation
The three math columns report Avg@32 / Pass@32; the two out-of-distribution columns report Avg@32. All scores are percentages; absolute gains are percentage points.
| Variant | AIME24 | AIME25 | HMMT25 | MBPP+ | GPQA_D |
|---|---|---|---|---|---|
| Olmo3-7B-TK Base | 71.88 / 93.33 | 61.77 / 90.00 | 41.98 / 73.33 | 64.55 | 38.86 |
| w/ OPSA (step 19) | 77.29 / 93.33 | 67.08 / 93.33 | 47.50 / 83.33 | 66.33 | 40.93 |
| Δ | +5.41 / +0.00 | +5.31 / +3.33 | +5.52 / +10.00 | +1.78 | +2.07 |
| Relative gain | +7.5% / +0.0% | +8.6% / +3.7% | +13.1% / +13.6% | +2.8% | +5.3% |
Results and gains above are reproduced from the authors' reported results table. GPQA_D denotes GPQA-Diamond. See the paper and project repository for experimental details.
Usage
Recommended sampling settings
After OPSA training, this model may benefit from a higher sampling temperature.
We recommend trying temperature=1.5 with do_sample=True, top_p=0.8, and
top_k=20 as a starting point for improved reasoning performance. The best
temperature depends on the task, so tune it on your own validation examples;
higher temperatures do not always yield better results.
Quick start
Use the checkpoint's chat template to format prompts for the Think model.
pip install "transformers>=4.57.0" accelerate torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Tuwhy/Olmo-3-7B-Think-OPSA"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
messages = [{"role": "user", "content": "Solve: 3x + 5 = 20."}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to(model.device)
outputs = model.generate(
**inputs,
do_sample=True,
temperature=1.5,
top_p=0.8,
top_k=20,
max_new_tokens=32768,
)
print(tokenizer.decode(
outputs[0][inputs.input_ids.shape[1]:],
skip_special_tokens=True,
))
The decoding settings above are an inference example, not a specification of the evaluation protocol for every result in the table.
License and limitations
This checkpoint is released under the Apache 2.0 license, following the base model. It can produce incorrect, biased, or harmful content; OPSA's reasoning results do not establish improved safety. For the original model's development details and limitations, see the base model card.
Citation
OPSA
If you find this work useful, please cite:
@article{ding2026does,
title={Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement},
author={Ding, Yi and Zhang, Ruqi},
journal={arXiv preprint arXiv:2608.31046},
year={2026}
}
Base model
Please also refer to the Olmo 3 model card for the original model citation.