ReadAll / ReadTwice 14B — Stage A step230
Full BF16 weights of a Qwen2.5-14B-Instruct model after 230 optimizer updates of Stage A supervised fine-tuning. This checkpoint contains no Stage B or RL updates. It was exported without additional training and is an intermediate checkpoint from a 286-update Stage A schedule.
Model and training
- Base:
Qwen/Qwen2.5-14B-Instruct, revisioncf98f3b3bbb457ad9e2bb7baf9a0125b6b88caa8. - Parameters: 14,770,033,664; BF16; original 32,768-token native context configuration.
- Source data: Xirui1208/readall-sft-stage-a-b, revision
877b52deb0622bed8f8102693f6907413df7d693. - Local quality revision: 73,400 training rows and 1,575 validation rows; 16 training rows were quarantined and 101 benchmark-overlap validation rows were removed.
- Global batch 256; learning rate 1e-5; warmup ratio 0.1; four GPUs; max training sequence length 12,288. Detailed settings and accumulation adaptation are in
training_metadata.json.
Loading
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Xirui1208/readall-readtwice-14b-stage-a-step230-20260923"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
The checkpoint supports the recurrent ReadAll / ReadTwice protocol: first SKIM every source chunk to construct an outline, then revisit every original chunk with the question, outline and evolving memory, and finally answer from memory. KEEP_MEMORY retains the preceding memory; other UPDATE text replaces it. Memory rewriting follows the original Stage A protocol.
The long-document results below require this external recurrent and paginated evaluation loop. They do not indicate a native 112K context window or a single generate call on the entire document.
Recorded evaluation
| Pinned HQA length | Questions | SubEM correct | SubEM | EM | Token F1 |
|---|---|---|---|---|---|
| 56K | 128 | 96 | 75.00% | 60.16% | 71.99% |
| 112K | 128 | 99 | 77.34% | 60.94% | 74.38% |
These are the existing completed evaluations of this exact Stage A step230 export. The same 128 questions are reused at both lengths. No completed 224K or LongBench score is available for this checkpoint.
Protocol: temperature 0, top_p 1, seed 801; all chunks read in both passes; lossless chunks of at most 5,000 tokens; SKIM/UPDATE/FINAL output budgets 512/1,024/1,024. Outlines are paginated with at most 24 sections per page and an 8,192-token navigation prompt budget; memory is carried serially between pages. Gold labels are used only for scoring. SubEM is normalized bidirectional substring matching, with strict yes/no/noanswer handling; it is distinct from exact match.
The tokenizer, model configuration and saved generation configuration are preserved from the export. The saved generation defaults differ from the benchmark settings, so set benchmark sampling options explicitly. See evaluation_protocol.json and evaluation_results.json.
Contents
Six safetensors weight shards, their index, tokenizer assets, model/generation configuration, training metadata and evaluation summaries. The original distributed optimizer checkpoint remains on the training machine and is not part of this inference release.