yarrow-forecast
yarrow-forecast forecasts U.S. party primary and nomination outcomes. Given a question, its resolution criteria, and the evidence available as of a stated date, one forward pass returns a calibrated probability in [0, 1].
The model does not write an answer and does not abstain. It reads the probability from the next-token distribution. There is nothing to parse. The weights run locally: 27B parameters, 256K context, no hosted API in the loop.
Why use it
The marginal cost is about zero. The weights sit on your own GPU. A forward pass does not incur a per-call fee.
One forward pass, no generated text. Measured batch inference is 0.12–1.6 seconds per question. There is no sampling and no token-by-token decode. The probability is read from the next-token distribution.
No third-party service. Scoring does not depend on an outside API, a proxy, or a rate limit. The model answered every item in the evaluation.
The weights can be trained further. A LoRA update followed by a merge left yes/no calibration error at 0.001–0.003. On 15 other checked items, the answers were unchanged after the merge.
Model details
| Model | yarrow-forecast |
| Parameters | 27B |
| Context length | 256K tokens |
| Task | P(the event resolves YES) |
| Validated domain | U.S. party primary and nomination elections |
| Output | One probability in [0, 1] |
| Inference | One forward pass, deterministic, 0% abstention |
| License | CC BY-NC-4.0 |
Input and output
| Question | The event to forecast |
| Resolution criteria | How a YES or NO outcome is decided |
| Evidence | News, polls, and other material available at prediction time |
| As-of date | The date the forecast is conditioned on |
| Output | p ∈ [0, 1] |
Usage
forecast.py loads the weights once. Each call returns only the calibrated probability.
from forecast import Forecaster
model = Forecaster("YarrowLab/yarrow-forecast")
p = model.forecast(
title="Will candidate X win the nomination?",
rules="Resolves YES if candidate X is declared the nominee.",
evidence="[E1] ...",
as_of="2026-09-05",
)
print(p) # 0.71
ps = model.forecast_batch([
{"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"},
{"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"},
])
Evaluation
Validated domain: U.S. party primary and nomination questions with settled outcomes. Same questions, same evidence, same outcomes. Each model answered in its own setup. yarrow-forecast answered all 150.
| Model | Brier ↓ | AUC ↑ | Directional accuracy | Answered |
|---|---|---|---|---|
| yarrow-forecast | 0.175 | 0.81 | 76.0% | 150 |
| Claude Sonnet 4.6 | 0.171 | 0.84 | 70.1% | 117 |
| GPT-5.5 | 0.185 | 0.87 | 65.5% | 148 |
| Historical base rate | 0.239 | 0.51 | 62.7% | 150 |
| GPT-5.4-mini | 0.242 | 0.51 | 62.4% | 149 |
| OpenForecaster-8B | 0.265 | 0.72 | 61.3% | 93 |
Directional accuracy on the full set is 76.0%, above Claude Sonnet 4.6 (70.1%) and GPT-5.5 (65.5%). Brier is 0.175, below GPT-5.5 (0.185), GPT-5.4-mini (0.242), and OpenForecaster-8B (0.265), and about 24% lower than the historical base rate (0.239). Across five out-of-sample fold splits the Brier stays in 0.175–0.185 and directional accuracy in 72.7–76.0%. Abstention is 0%.
Claude Sonnet 4.6 scores 0.171 Brier on the 117 questions it returned. OpenForecaster-8B's 93 questions are the ones it returned. Unanswered items were left unanswered.
A later set of 166 settled questions from the same category scores 0.1653 Brier (95% interval 0.136–0.200, clustered by election night).
With the question and the resolution rule alone, AUC is about 0.51. With the evidence supplied, AUC is 0.808.
License
CC BY-NC-4.0. Attribution is required. Commercial use is not permitted.