Qwen3-1.7B · ScienceWorld · trajectory SFT on gpt-5.4-mini rollouts (3 epochs)

Qwen3-1.7B fine-tuned on full-episode ReAct trajectories that gpt-5.4-mini (OpenAI API, reasoning_effort=medium) played on the 2059 ScienceWorld training task variations (one episode each, at most 30 turns, teacher success 41.96%). This is the SFT arm of an SFT-vs-online-ROSE comparison; the ROSE arm is SeanWang0027/qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-3ep.

Training

• Data: 2059 episodes, 46035 supervised teacher turns. Nothing filtered or masked: failed episodes and turns the simulator rejected are included.
• One training row per episode. Cross-entropy only on each teacher reply and its <|im_end|>; observations, the instruction and the assistant header are context only. Only the teacher's visible reply is used (no hidden reasoning).
• 3 epochs (192 steps), batch 32, AdamW lr 1e-5, weight decay 0, cosine schedule with 10% warmup, fp32 training, max length 8192. This is the epoch-3 checkpoint, stored in bf16.

Evaluation

ScienceWorld test split (200 task variations, AgentGym protocol), 4 independent passes, temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn. Success = final score 100. Mean ± std across the 4 passes.

ROSE minus SFT, paired bootstrap over tasks: +4.62 points success [0.75, 8.75], +0.139 Avg@1 [0.072, 0.203].

Known limitation

The teacher copied the prompt's action template open/close OBJ literally in 29.89% of its turns (the simulator rejects it). This model reproduces that on 74.88% of its turns and rarely escapes the loop; episodes without it succeed 47.71% of the time. Fixing the action list in the prompt and re-collecting would likely help more than any training change.

Prompt format

ReAct. The conversation is AgentGym's ScienceWorld instruction as a user turn, a canned assistant acknowledgement, then one user turn per observation. The model replies Thought:\n...\n\nAction:\n<one command>. Render with Qwen3's chat template with enable_thinking=False.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(REPO); model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="bfloat16")
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")

All checkpoints of this study (ScienceWorld test, 200 task variations)

Base Qwen3-1.7B scores 0.12%. Every number is the mean ± std of 4 independent passes at temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn; success = final score 100, Avg@1 = mean final score / 100.

How to read this table:

• The teacher (gpt-5.4-mini, medium) copies the prompt's action template open/close OBJ literally in 29.89% of its own turns; the simulator rejects it every time. Students inherit the habit, and their test success tracks how strong the habit is in the particular checkpoint (last column). In online ROSE the habit oscillates during training, so single-checkpoint numbers are noisy: ROSE epoch 2 was saved at a relapse peak, epochs 3 and 5 near minima, epoch 6 during another relapse.
• Re-evaluating the same checkpoint (ROSE epoch 3) with 4 fresh passes moved its success rate from 17.50% to 15.38%, so differences under about 3 points are within evaluation noise.
• SFT epochs 1–2 are mid-schedule checkpoints of one 3-epoch run with a cosine learning-rate schedule; only epoch 3 saw the full decay.
• ROSE epochs 4–6 continue from the epoch-3 weights with a freshly initialised AdamW (no optimizer state had been saved).