92M Block-MTP — step 10000
Condition: 92m-blktrunc-lr3e3-wu05-s42. This branch contains ready-to-load model weights,
config, tokenizer and the exact shared-core implementation. Each checkpoint
has its own step-N branch. Install the packages in requirements.txt.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "violetxi/hparam-92m-block-mtp-truncated-peak_lr-3e-3-wu-0p05-s42"
revision = "step-10000"
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, revision=revision, trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda" if torch.cuda.is_available() else "cpu").eval()
inputs = tokenizer("Pe2e4 Pe7e5 Ng1f3", return_tensors="pt").to(model.device)
generated = model.generate(**inputs, max_new_tokens=20, do_sample=True,
temperature=1.0, top_k=0, top_p=1.0)
print(tokenizer.decode(generated[0], skip_special_tokens=True))
Evaluation semantics
Three recurrent passes score three fixed, causally ordered query slots.
generate() commits that three-token block before the next recurrent block;
the mask token (ID 81) cannot be generated. Sampling and greedy generation,
padding and num_return_sequences are supported; KV caching and beam search
are not supported. Context is 1,024 logical tokens including the query block.
model.block_logits(input_ids) returns [batch, 1, 3, vocab] for the next block.
model(input_ids, labels=input_ids) uses the registered dense non-overlapping
block loss: logits[:, t] scores token t+1, with every group of three seeing
only the prefix before that group. It is not ordinary teacher-forced NTP.
Benchmark inference is not performed by publication. Condition selection
remains governed by the sweep's validation and collapse rules.
Safetensors values preserve the native model tensors exactly. Optimizer and
RNG recovery state remains in the native archive; it is not needed for HF
inference. training_summary.json binds this export to the native checksum
and source versions. Adapted core code licenses are included alongside it.