TR-HASH MoE 100M - Agentic Refinement
Status: stopped and evaluated at optimizer update 9,791. The selected checkpoint is published at the repository root. The intended 70B-token schedule was stopped after 38.50B refinement tokens because the historical PIQA
acc_normgain remained limited to +0.16 percentage point over the pretraining final. This is an evaluated intermediate refinement checkpoint, not a completed 70B pass.
This run restarts from the verified final/ weights of TR-HASH MoE 100M - 125B Agentic Pretraining. It loads model weights only and creates a fresh optimizer and learning-rate schedule.
Root checkpoint
| Field | Value |
|---|---|
| Selected checkpoint | token_pack_011_9791 |
| Optimizer updates | 9,791 / 17,801 |
| Refinement-token exposure | 38,499,778,560 |
| Total source-token exposure | 163,499,376,640 |
| Learning rate at checkpoint | 4.8412e-5 |
| Parameters | 100,366,720 |
| Root weight file | model.safetensors |
| Weight SHA-256 | 9a7935ea8a213bb578b9932eb6905ef1bbe907940483712ce0ce8b707c32a3ea |
The repository root also contains the model configuration and its native 32K
Agentic tokenizer. The token_pack_* directories remain available as resumable
training checkpoints with optimizer state.
Evaluation
PIQA was evaluated over all 1,838 validation examples with the historical MLX
scorer used for this lineage: goal + " " + solution.lstrip(), the model's own
tokenizer, add_special_tokens=False, no BOS/EOS, and no chat template.
| Checkpoint | PIQA acc | PIQA acc_norm |
|---|---|---|
| Pretraining final | 64.69% | 65.51% |
| Corrected refinement step 9,791 | 65.45% | 65.67% |
| Change | +0.76 pp | +0.16 pp |
Machine-readable PIQA results and the exact scorer, tokenizer, and checkpoint
hashes are provided in eval_results.json.
Corrected optimizer-update contract
| Setting | Corrected value |
|---|---|
| Source data | same audited 70B unique-token core |
| Replay | none |
| Parameters | full model |
| Tokens per optimizer update | 3,932,160 |
| Planned optimizer updates | 17,801 |
| Scheduled token exposure | 69,996,380,160 |
| Unused audited tail | 3,407,872 tokens |
| Optimizer | fresh AdamW |
| Peak learning rate | 1e-4 |
| Scheduler | cosine |
| Warmup | 500M tokens |
| Weight decay | 0.1 |
| Sequence length | 2,048 |
| Intended hardware | 8 x NVIDIA RTX 5090 |
| Batch per GPU | 16 sequences |
| Gradient accumulation | 15 |
The Complexity framework now prints an OPTIMIZER UPDATE CONTRACT block before data loading and rejects a 100M refinement launch if LR, scheduler, warmup, weight decay, or tokens per update differ from this contract. Runtime metrics retain the historical step fields and also expose optimizer_update, optimizer_updates_planned, and tokens_per_optimizer_update.
Provenance note
The earlier 100M refinement archive was withdrawn because it used 229,376 tokens per optimizer update, causing many more AdamW updates per source token. It must not be used as the initialization point for this run.
License
Released under CC BY-NC 4.0. Source datasets retain their own licenses and terms.