AIME quality estimator, VibeThinker-1.5B
A binary accept/route classifier for a two-tier LLM cascade on AIME. It reads a
VibeThinker-1.5B answer and predicts whether that answer is correct. A route
prediction means escalate the query to a stronger model.
Trained for CRE-Router on
ymoslem/AIME-24-25-26-router.
The generator is named on purpose. This estimator is trained on VibeThinker-1.5B outputs and should be applied to VibeThinker-1.5B outputs. It does not transfer to a different efficient tier without being retrained.
Input format
Build the input exactly as it was trained, or the scores will not mean what they should:
text = f"{question} {tokenizer.sep_token} {last_1000_words_of_output} "\
f"{tokenizer.sep_token} {num_output_tokens}"
Keep only the last 1,000 words of the model's output, then truncate at 4,096
tokens. Class 0 is route, class 1 is accept.
Results
On the 450 rows of test_vibethinker, 90 problems from AIME 2024 to 2026 scored
across 5 sampled runs:
| Class | F1 | Precision | Recall |
|---|---|---|---|
| Route | 0.779 | 0.789 | 0.769 |
| Accept | 0.924 | 0.920 | 0.928 |
| Macro | 0.851 |
Accuracy 0.887. It escalates 114 of 450, catching 90 of the 117 wrong answers and wrongly escalating 24 correct ones.
Two caveats worth reading before using these numbers
The evaluation split is also the early-stopping split. This is deliberate, so the checkpoint stays comparable with the earlier estimators in this project, but it means 0.851 is a selection criterion rather than a held-out score.
Much of the signal is output length, and a length rule does as well. The input string ends with the token count, and the predicted accept probability correlates $-0.77$ with it. Composing the actual cascade, VibeThinker-1.5B escalating to Qwen3-30B-A3B-Thinking, and matching the number of escalations:
| Escalation rule | Escalated | Cascade accuracy |
|---|---|---|
| none, always VibeThinker | 0 | 74.0% |
| random | 115 | 76.7% |
| this classifier | 114 | 83.1% |
| longest answers first | 115 | 84.1% |
| all, always Qwen3-30B | 450 | 85.8% |
Selecting clearly beats not selecting: against random escalation of the same count the classifier gains 6.4 pp. But it does not beat sorting by output length, and the difference between them, $+0.98$ pp for the length rule, has a 95% interval of $[-0.98, +3.47]$ that spans zero. Report the length rule as the baseline, not random.
Training
ModernBERT-base, 10 epochs with early stopping at patience 4, best epoch 3, learning rate 5e-5, effective batch 64 (16 with 4 accumulation steps), class weights for the 86% accept majority, on one A100.
Licence
CC BY-NC-SA 4.0, inherited from the training data, whose AIME 2026 source carries non-commercial share-alike terms. The base model is Apache-2.0.