ZeroHero Craftax strong: BATS / LinUCB, cycle 300
A 12,969,600-parameter language executor for Craftax-Dialogue hidden-goal navigation. This uses the Qwen3 architecture, trained from random initialization; it does not inherit pretrained Qwen3 weights. The student first received supervised training on expert trajectories, then started online training from SFT checkpoint 400 (the strong initialization). This is the final cycle-300 policy, not a checkpoint selected by the final evaluation score.
Model and task
- Six transformer layers; hidden size 384; intermediate size 1264.
- Six attention heads, three key/value heads; 2,048-token context.
- Custom 4,096-token tokenizer; weights stored in FP32.
- The executor observes a local symbolic map and dialogue, then emits reasoning, an action, and an optional question. The hidden navigation target is available to the oracle, not directly to the executor.
- Gemma supplies public-state action labels and separately answers questions using privileged state. Reasoning is retained in the response format.
- This is a specialized executor, not a general chat model or a standalone oracle.
Training
Method: BATS / LinUCB. Training seed: 93001. Completed cycles: 300. Recorded training episodes: 20224. Recorded training decisions: 622844.
DAgger uses expert labels on student-visited public states, replay and SFT anchors.
LinUCB adaptively selects PPO or DAgger updates from controller feedback.
The executed scalar settings are in training_config.json.
Historical learner source: f4e12de02ee41556504c4536a785e4de6be4a735.
Gemma teacher revision: 4d7ae4984b7db7de8f8457170b3f1a419ee76d52.
Packaged experiment code and environment interfaces:
ZeroHero, iclr2027 branch.
Matched final evaluation
| Policy | Successes | Success rate |
|---|---|---|
| This model | 458/512 | 89.45% |
| Gemma action expert + Gemma question oracle | 425/512 | 83.01% |
Mean episode length: 29.26 turns. All 512 evaluation seeds and hidden targets match the expert comparison. No target repairs were applied. Students sample at temperature 1; expert actions use temperature 0. The turn limit is 75 and executor response limit is 256 tokens. Both agents use the private Gemma question oracle; answers need not be identical between runs. The expert evaluation counts malformed actions as invalid turns, without repairing the action. These are single-training-seed results, not an estimate of variability across training seeds or vanilla Craftax achievement scores.
Load and generate
Requires Transformers with Qwen3 support (checked with the learner runtime).
Use the included raw prompt format; this tokenizer has no generic chat template.
example_prompt.txt is one public initial prompt from the final evaluation.
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from pathlib import Path
repo_id = "RLMM/zerohero-craftax-qwen3-13m-strong-linucb"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto").eval()
prompt = Path(hf_hub_download(repo_id, "example_prompt.txt")).read_text()
inputs = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False)
output = model.generate(
**inputs, do_sample=True, temperature=1.0, top_k=0, top_p=1.0,
max_new_tokens=256, pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
This demonstrates one response. Interactive evaluation additionally needs the
ZeroHero environment wrapper, observation formatter, action parser and Gemma oracle.
value_head.pt contains the separate value-head weights used by the training code;
standard AutoModelForCausalLM loads only the language policy. Optimizers, RNG,
replay and bandit recovery state are not part of this inference release.
sha256.json records file hashes. See evaluation.json and provenance.json for
the evaluation and original checkpoint hashes.