StraTA / Qwen2.5-1.5B-Instruct / ALFWorld — optimizer step 10

Reproduction of StraTA (Strategic Trajectory Abstraction, arXiv:2605.06642, code xxyQwQ/StraTA @ 592238b) on ALFWorld with Qwen/Qwen2.5-1.5B-Instruct.

This checkpoint: optimizer step 10 of 150.

Training curve

Validation success rate on the full ALFWorld valid_seen split (140 tasks), measured every 10 optimizer steps during training. Step 0 is the untrained base model.

Final evaluation (step 150, 2 seeds)

Run with evaluate-strata-alfworld-text-qwen2.5-1.5b.sh: val_only, 140 tasks, temperature 0.7 / top_p 0.8 / top_k 20, max 50 env steps.

Auxiliary metrics: action format_rate 1.000 (both seeds), mean episode length 13.2–14.2 steps (base model: 49.7).

Hyperparameters

All algorithm hyperparameters follow the paper: 150 steps, batch size 16, 4 strategies per task, 8 rollouts per strategy, oversampling ratio sigma=8, aggregation ratio delta=0.5, length penalty threshold lambda=0.5, self-judgment reward weight kappa=0.1.

Infrastructure deltas

This run reproduces the upstream 7B recipe on 2x A100 80GB. A full diff of the resolved Hydra config against train-strata-alfworld-text-qwen2.5-7b.sh shows 6 differing keys out of 533, all infrastructure -- no algorithm or optimization parameter was changed:

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "chanyoungkim/strata-qwen2.5-1.5b-alfworld-step010"
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
tok = AutoTokenizer.from_pretrained(repo)

The model expects the StraTA ALFWorld prompt format: it first emits a plan inside <strategy>...</strategy>, then acts, emitting one action per turn inside <action>...</action>.

Files

training_state/ lets you resume RL training from this exact optimizer step. Note the shards are written for world_size=2 (2 GPUs, FSDP). Resuming on a different number of GPUs requires resharding. To resume, place the files back as checkpoints/<exp>/models/global_step_10/actor/ and point verl at that directory.

The published model.safetensors is a bf16 cast of the fp32 master weights; the original fp32 tensors are preserved inside training_state/.

All checkpoints from this run

step 10 · step 20 · step 30 · step 40 · step 50 · step 60 · step 70 · step 80 · step 90 · step 100 · step 110 · step 120 · step 130 · step 140 · step 150

Caveats

• Do not use is_correct or pass@1 from the logs. Both are hardcoded to 1.0 in the workflow and are meaningless. The correct metric is val/average/metrics/success_rate.
• Evaluation uses the StraTA inference protocol (one strategy, then one trajectory conditioned on it; no diversity selection, no self-judgment). This is what strata_workflow.run() does when rollout_engine.validate is set, and it matches evaluate-strata-*.sh.
• Numbers here are not directly comparable to models evaluated under a different agent scaffold (different prompt format, context window, or sampling temperature).
• The evaluation set is ALFWorld valid_seen (140 tasks, all of them -- not a subsample). valid_unseen was not evaluated.

Citation

@article{xue2026strata,
  title={StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction},
  author={Xue, Xiangyuan and Zhou, Yifan and Wang, Zidong and Tang, Shengji and Torr, Philip and Ouyang, Wanli and Bai, Lei and Yin, Zhenfei},
  journal={arXiv preprint arXiv:2605.06642},
  year={2026}
}