Qwen3-1.7B · ALFWorld · online ROSE 10+5 with a gpt-5.4-mini teacher — epoch 2 of 3

Student: Qwen/Qwen3-1.7B. Part of a study comparing teacher-trajectory SFT, online multi-turn ROSE and on-policy distillation for multi-turn agents.

Training

Online multi-turn ROSE: the student plays the first 10 turns in a live environment, the teacher continues the same environment for up to 5 turns, and only the teacher's turns are cross-entropy targets. 32 episodes per step, AdamW constant lr 1e-5, bf16, thinking off, 512 tokens per turn. Teacher: gpt-5.4-mini (OpenAI API, reasoning_effort=medium; only the visible reply is used). 2181 training games, 68 steps per epoch, 3 epochs = 204 steps; this is step 136. About 7% of steps were empty because the ALFWorld env server had to be recycled (disk leak).

Evaluation

ALFWorld test split (AgentGym, 200 games), avg of 4 passes, temperature 0.4, 512 tokens/turn, 30 rounds, thinking off. Success 42.00% ± 1.27. Epochs 1/2/3: 32.00 / 42.00 / 40.25. Same-teacher trajectory SFT after 3 epochs: 58.25% (SFT − ROSE = +18.00, 95% CI [12.75, 23.75]). Reference points on the same protocol: base Qwen3-1.7B 19.38%, Qwen3-32B itself 52.12%.

Prompt format

ReAct, multi-turn chat: the environment's AgentGym instruction as the first user turn, a canned assistant acknowledgement, then one user turn per observation; the model replies Thought:\n...\n\nAction:\n<one action>. Render with Qwen3's chat template and enable_thinking=False.