jin10-sdft-qwen2.5-7b
Qwen2.5-7B-Instruct after twelve windows of rolling self-distillation fine-tuning (SDFT) on Jin10 prediction-market data. Given a market, its recent hourly prices and a pool of macro-wire headlines, it predicts how far the next hour's move will land above or below a mean-reversion baseline — not the price itself.
Task
The baseline is a frozen OLS fit on the current price,
0.064650 − 0.031089 × price, estimated once on the warmup window's 461 training
rows and held fixed for the whole schedule. The model forecasts the residual
against it, discretised into nine quantile buckets.
Input is a system message defining the buckets plus a user message with the market description, ~23 hourly price points and ~30 headlines. Output is a reasoning chain and one bucket:
<reasoning>
...which headlines, if any, actually bear on this market...
</reasoning>
<answer>
B4
</answer>
For scoring the bucket becomes a number as anchor_baseline + midpoint(bucket),
with midpoints [-0.153, -0.085, -0.026, -0.011, +0.000, +0.014, +0.034, +0.059, +0.166].
Results
6323 forward rows, each scored by the checkpoint that had not yet trained on that
window. Skill is 1 − MSE(model)/MSE(anchor); abstentions fall back to the anchor.
| skill | Pearson | direction | bucket acc | abstain | |
|---|---|---|---|---|---|
| this model | +0.0270 | 0.175 | 0.724 | 0.111 | 37 / 6323 |
| claude-sonnet-5 (zero-shot) | +0.0254 | 0.237 | 0.749 | 0.133 | 0 |
| frozen OLS anchor | 0.0000 | 0.149 | 0.718 | — | — |
| codex-gpt-5.6 (zero-shot) | −0.0176 | 0.263 | 0.735 | 0.166 | 0 |
| base Qwen2.5-7B-Instruct | −0.0414 | 0.051 | 0.711 | 0.101 | 2 |
On the 2145 permanent holdout rows the model never trains on: +0.0251, the only model in the comparison clearly ahead of the anchor there (runner-up +0.0079).
What this model does not do
It has not learned to classify. Bucket accuracy is 0.111 against a nine-bucket chance rate of 0.111. It beats the anchor because being one bucket off still moves the prediction the right way, not because it picks the right bucket.
It barely moves. Predicted standard deviation is 0.0203 against a true 0.1038 — about a fifth as much as the target varies. Inspecting the predictions: it selects the near-zero bucket B4 on 78.7% of rows, where the offset is +0.00007, so on four rows in five the prediction is the anchor. It never selects the extreme buckets B0 or B8 at all, so the largest moves in the data are outside the range it can express. All of the squared-error improvement comes from the 21% of rows where it does move, and on those it is closer than the anchor 54.9% of the time.
Two zero-shot frontier models understand the task better than it does — higher Pearson, higher bucket accuracy — and still score worse, because they are badly calibrated in magnitude. This model wins on restraint, not on insight.
The rolling schedule is not what earns the result. Frozen after the warmup window alone — 461 rows, 9.5% of the training data — the model already scores +0.0170. The remaining eleven windows and 4378 rows add +0.0024. Freezing costs nothing over time either: the warmup-only checkpoint shows no decay when evaluated on windows up to ten months past its freeze point.
Training
| base | Qwen2.5-7B-Instruct |
| protocol | 60-day warmup, 30-day windows, 70% train / 30% permanent holdout per window |
| demonstrations | 4839 chains written by the student's own 7B weights, shown the realised move |
| objective | on-policy SDFT — the student samples, a teacher (same weights, prompted with a worked example) scores those tokens, loss is the per-token KL |
| teacher | reloaded from the base model each window, mixed toward the student at 0.01 per step inside it |
num_loss_tokens_to_skip | 0 |
| learning rate | 5e-5 warmup, 5e-6 thereafter, cosine |
| hardware | 8 × A100-40GB, FSDP FULL_SHARD |
| optimizer updates | 172 across twelve windows |
Two settings matter and both were established by ablation. Taking the teacher from the previous window's checkpoint instead of the base model collapses the run — completion length falls from 609 tokens to 6 and the model abstains on every probe from the fifth window on. Freezing the teacher rigidly instead of resyncing keeps the loss at 0.19–0.20 every window instead of falling to 0.09, and damages the model's off-task behaviour from the first window.
Off-task behaviour
Twelve prompts unrelated to the task, greedy decoding, across all twelve
checkpoints: generation length stays at or above the base model's 276 tokens with
no downward trend. The cost is format — 8 to 11 of 12 off-task answers come back
wrapped in <reasoning> tags, with normal correct prose inside. A variant at
learning rate 1e-5 leaks nothing at all but its skill falls to +0.0006.
Limitations
- Evaluated on one market family over one twelve-month span. No claim outside it.
- Bucket accuracy and RMSE here are forecasting metrics, not trading P&L.
- The forward cells are prequential: the model has seen none of a window when scored on it, but 70% of those rows are trained on immediately afterwards. The 2145 holdout rows are never trained on and give +0.0251.
- The comparison against zero-shot models is not like-for-like: this model is constrained to nine bucket midpoints, they are free to emit any number.