Qwen3-1.7B · ScienceWorld · online multi-turn ROSE 10+5 with a gpt-5.4-mini teacher (3 epochs)

Qwen3-1.7B trained with online multi-turn ROSE: in every episode the student plays the first 10 turns in a live ScienceWorld environment, then the teacher (gpt-5.4-mini, OpenAI API, reasoning_effort=medium) continues the same environment for up to 5 turns. Only the teacher's turns are training targets (cross-entropy); the student's own turns and all observations are context. So the student is supervised on states it reaches itself. The same-teacher SFT arm is SeanWang0027/qwen3-1.7b-sciworld-sft-gpt54mini-3ep.

Training

• 2059 ScienceWorld training task variations, 32 episodes per step, one episode per task per epoch, 3 epochs = 192 steps; fresh student prefixes and teacher continuations every epoch (24390 teacher API calls in total).
• AdamW, constant lr 1e-5, bf16, thinking off, 512 tokens per turn. Only the teacher's visible reply is used (no hidden reasoning). This is the epoch-3 checkpoint (step 192).

Evaluation

ScienceWorld test split (200 task variations, AgentGym protocol), 4 independent passes, temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn. Success = final score 100. Mean ± std across the 4 passes.

ROSE minus SFT, paired bootstrap over tasks: +4.62 points success [0.75, 8.75], +0.139 Avg@1 [0.072, 0.203].

A separate 1-epoch run of the same setup scored 8.88%. On the training rollouts the teacher raised the episode score by +0.112 on average over its 5 turns.

Known limitation

The teacher sometimes copies the prompt's action template open/close OBJ literally (the simulator rejects it); this model emits it on 35.22% of its turns, against 74.88% for the SFT arm.

Prompt format

ReAct. The conversation is AgentGym's ScienceWorld instruction as a user turn, a canned assistant acknowledgement, then one user turn per observation. The model replies Thought:\n...\n\nAction:\n<one command>. Render with Qwen3's chat template with enable_thinking=False.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(REPO); model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="bfloat16")
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")

All checkpoints of this study (ScienceWorld test, 200 task variations)

Base Qwen3-1.7B scores 0.12%. Every number is the mean ± std of 4 independent passes at temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn; success = final score 100, Avg@1 = mean final score / 100.

How to read this table:

• The teacher (gpt-5.4-mini, medium) copies the prompt's action template open/close OBJ literally in 29.89% of its own turns; the simulator rejects it every time. Students inherit the habit, and their test success tracks how strong the habit is in the particular checkpoint (last column). In online ROSE the habit oscillates during training, so single-checkpoint numbers are noisy: ROSE epoch 2 was saved at a relapse peak, epochs 3 and 5 near minima, epoch 6 during another relapse.
• Re-evaluating the same checkpoint (ROSE epoch 3) with 4 fresh passes moved its success rate from 17.50% to 15.38%, so differences under about 3 points are within evaluation noise.
• SFT epochs 1–2 are mid-schedule checkpoints of one 3-epoch run with a cosine learning-rate schedule; only epoch 3 saw the full decay.
• ROSE epochs 4–6 continue from the epoch-3 weights with a freshly initialised AdamW (no optimizer state had been saved).