Laplace v3.7 Qwen3-14B — hindsight-guided (epoch 1.00)
Qwen3-14B supervised-fine-tuned for binary-event forecasting from a question,
resolution criteria, forecast date, and chronologically available retrieved evidence.
This arm uses the dataset's hindsight_guided reasoning and forecast targets.
Checkpoint: step 528, epoch 1.004 of 2 (526 steps/epoch). The full run ended at step 1,052.
Results
Per-question results on 304 questions comprising 1,144 forecast-instance rows, with
8 generations per row (9,152 generations). This run and evaluation used the
pre-revision labels and
artifacts/sft/laplace-v37/test_questions.jsonl. The base rate is 0.5921, so
UNC = pi(1-pi) = 0.2415.
| export | Brier ↓ | REL ↓ | RES ↑ | UNC | ECE ↓ | AUROC ↑ | parse rate |
|---|---|---|---|---|---|---|---|
| step 528 (this repo) | 0.1888 | 0.0237 | 0.0755 | 0.2415 | 0.1324 | 0.8174 | 100.0% |
| final root, step 1,052 | 0.1894 | 0.0197 | 0.0680 | 0.2415 | 0.1291 | 0.8090 | 100.0% |
BS = REL - RES + UNC up to the reported finite-bin decomposition residual; REL/RES
use 10 equal-width bins. Coverage is complete: 304/304 questions and 1,144/1,144 rows.
Selection caveat. This checkpoint was selected retrospectively from test-set forecasting results. There was no forecasting validation set or forecasting early-stopping rule; validation during training measured language-model loss only. Its improvement over the final export is therefore observed on this test set, not shown to generalize. The Brier difference from the final export was not statistically significant.
Usage and prompt construction
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "snupilab/laplace-v37-qwen3-14b-hindsight-ep1.00"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
user = (
f"Question:\n{question}\n\n"
f"Resolution criteria:\n{resolution_criteria}\n\n"
f"Forecast date:\n{forecast_date}\n\n"
f"Retrieved information:\n{information}"
)
prompt = tok.apply_chat_template(
[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user},
],
tokenize=False,
add_generation_prompt=True,
enable_thinking=True,
)
The prompt layout is load-bearing and matches training and evaluation byte-for-byte. Use the dataset's forecasting system prompt and do not expose evidence published after the forecast date.
Probability parsing
Split on the last </think> and parse only the answer region. The strict preferred
forms are an <answer> tag containing a percentage or decimal, followed by named
probability/forecast/prediction/chance percentage or decimal forms. Last-percent
and last-decimal matches are diagnostic fallbacks. Values must lie in [0,1]; parse
failures were scored as Brier 1.0.
region = text.rsplit("</think>", 1)[-1].strip()
m = re.search(r"<answer>\s*([0-9]*\.?[0-9]+)\s*</answer>", region, re.I | re.S)
probability = float(m.group(1)) if m else None
Training
- Training data:
snupilab/laplaces-demon-forecasting-traces_v2, a private dataset licensed CC BY-SA 4.0. - Base model:
Qwen/Qwen3-14B, Apache-2.0. - Recipe: hindsight-guided reasoning and targets from the pre-revision dataset.
- Two epochs, seed 0, effective batch size 16, maximum sequence length 8,192.
- Training used DeepSpeed ZeRO-2 with CPU optimizer offload on 2 GPUs. DeepSpeed retained fp32 master weights and fp32 Adam moments; cross-size comparisons therefore also differ in optimizer precision from the 8B DDP runs.
Evaluation protocol
Frozen full-evidence evaluation: k=8, temperature=0.6, top_p=1.0, top_k=-1,
seed=1, max_tokens=8192, max_model_len=12288, n_bins=10. All 9,152
generations parsed strictly; truncation was 0.0%.
These scores are not directly comparable to either 8B sibling, because both 8B
cards use corrected labels from revision e5a27da6fa65, while this card reports the
pre-revision label set.