Qwen3-1.7B · ScienceWorld · trajectory SFT on gpt-5.4-mini rollouts — epoch 2 of 3
Intermediate checkpoint after the second epoch (step 128): 13.25% success, statistically the same as epoch 3 (+0.38 points, 95% CI [-2.12, 2.88]).
Training
- Data: 2059 full-episode ReAct trajectories played by
gpt-5.4-mini(OpenAI API,reasoning_effort=medium) on the ScienceWorld training task variations, one episode each, at most 30 turns, teacher success 41.96%; 46035 supervised teacher turns. Nothing filtered or masked: failed episodes and rejected actions are included. Only the teacher's visible reply is used. - One row per episode; cross-entropy only on each teacher reply and its
<|im_end|>. - One 3-epoch run (192 steps): batch 32, AdamW lr 1e-5, weight decay 0, cosine schedule with 10% warmup over all 192 steps, fp32 training, max length 8192. This checkpoint is step 128, stored in bf16.
All checkpoints of this study (ScienceWorld test, 200 task variations)
Base Qwen3-1.7B scores 0.12%. Every number is the mean ± std of 4 independent passes at temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn; success = final score 100, Avg@1 = mean final score / 100.
| method | epoch | success rate (avg of 4 passes) | Avg@1 | share of turns that are the literal open/close … | model |
|---|---|---|---|---|---|
| SFT | 1 | not evaluated | — | — | qwen3-1.7b-sciworld-sft-gpt54mini-ep1 |
| SFT | 2 | 13.25% ± 2.56 | 0.1574 | 74.20% | qwen3-1.7b-sciworld-sft-gpt54mini-ep2 ← this model |
| SFT | 3 | 12.88% ± 1.24 | 0.1557 | 74.88% | qwen3-1.7b-sciworld-sft-gpt54mini-3ep |
| ROSE 10+5 | 1 | not evaluated | — | — | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep1 |
| ROSE 10+5 | 2 | 7.75% ± 1.92 | 0.1455 | 62.68% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep2 |
| ROSE 10+5 | 3 | 17.50% ± 2.29 (re-run: 15.38% ± 1.43) | 0.2942 (0.2512) | 35.22% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-3ep |
| ROSE 10+5 | 4 | 20.25% ± 2.25 | 0.3433 | 17.98% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep4 |
| ROSE 10+5 | 5 | 25.75% ± 1.30 | 0.3692 | 2.77% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep5 |
| ROSE 10+5 | 6 | 20.75% ± 1.82 | 0.2946 | 26.59% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep6 |
How to read this table:
- The teacher (
gpt-5.4-mini, medium) copies the prompt's action templateopen/close OBJliterally in 29.89% of its own turns; the simulator rejects it every time. Students inherit the habit, and their test success tracks how strong the habit is in the particular checkpoint (last column). In online ROSE the habit oscillates during training, so single-checkpoint numbers are noisy: ROSE epoch 2 was saved at a relapse peak, epochs 3 and 5 near minima, epoch 6 during another relapse. - Re-evaluating the same checkpoint (ROSE epoch 3) with 4 fresh passes moved its success rate from 17.50% to 15.38%, so differences under about 3 points are within evaluation noise.
- SFT epochs 1–2 are mid-schedule checkpoints of one 3-epoch run with a cosine learning-rate schedule; only epoch 3 saw the full decay.
- ROSE epochs 4–6 continue from the epoch-3 weights with a freshly initialised AdamW (no optimizer state had been saved).
Prompt format
ReAct. AgentGym's ScienceWorld instruction as a user turn, a canned assistant acknowledgement, then one user turn per observation;
the model replies Thought:\n...\n\nAction:\n<one command>. Render with Qwen3's chat template and enable_thinking=False.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(REPO); model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="bfloat16")
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")