ReadAll / ReadTwice RL step5 (initialized from step36)
This is the exact merged BF16 actor checkpoint at global step 5 of
readall_sft_hqa_rl_step36_memagent_readtwice_20260921_r1.
It continues training from
Xirui1208/readall-readtwice-step36-20260920.
The step number is local to this new training run.
Four safetensors shards, the weight index, tokenizer, and original model/generation
configuration are included. See checkpoint_manifest.json for SHA-256 hashes.
Greedy evaluation
Use greedy decoding: temperature=0, top_p=1 (do_sample=False in Transformers).
The original generation_config.json contains sampling defaults and is preserved
unchanged; explicitly override them for the greedy comparison.
| Metric | Initialization step36 | This step5 |
|---|---|---|
| HQA 224k EM | 57.81% | 57.81% |
| HQA 224k F1 | 72.15% | 73.19% |
| HQA 224k bidirectional SubEM | 74.22% | 75.78% |
| LongBench QA EM | 32.20% | 32.90% |
| LongBench QA F1 | 44.27% | 44.75% |
| LongBench QA custom bidirectional SubEM | 42.40% | 42.70% |
HQA covers all 128 questions with zero recorded evaluation error events. LongBench QA covers all 1,000 questions: 200 each from NarrativeQA, Qasper, HotpotQA, 2WikiMQA, and MuSiQue. One NarrativeQA final answer hit its output cap and remains in the denominator at zero. No API failures were recorded.
LongBench custom SubEM is not an official LongBench metric. It uses HQA-style
normalization and accepts a nonempty prediction if it contains a nonempty gold
answer or vice versa, taking the maximum over references. Invalid or empty
answers score zero. See evaluation.json for exact definitions and counts.
These are fixed development-set results and the gains are small; they do not establish a stable improvement on an unseen test set.
The HQA 224k result requires the matching paged ReadTwice pipeline: balanced
page-local outlines, 5,000-token chunks, 24 sections per page, serial memory
updates with <KEEP_MEMORY>, UPDATE cap 1,024, and seed 801. The model's native
context is 32,768 tokens; 224k is processed recurrently, not in one model call.
Direct full-context chat uses a different setup.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Xirui1208/readall-readtwice-step5-20260921"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id, torch_dtype="bfloat16", device_map="auto"
)
# Explicitly pass do_sample=False when generating for greedy evaluation.
This repository is public and can be downloaded without authentication.