Qwen3-1.7B · ScienceWorld · online multi-turn ROSE 10+5 with a gpt-5.4-mini teacher — epoch 5
Second epoch of the continuation (overall epoch 5): 25.75% success, the best checkpoint of the study — +8.25 points over epoch 3 (95% CI [4.75, 12.00]) and +12.88 over same-teacher SFT (95% CI [8.50, 17.62]). The open/close habit is almost gone in this checkpoint (2.77% of test turns).
Training
- Online multi-turn ROSE: in every episode the student plays the first 10 turns in a live ScienceWorld environment, then the
teacher (
gpt-5.4-mini, OpenAI API,reasoning_effort=medium) continues the same environment for up to 5 turns. Only the teacher's turns are training targets (cross-entropy); the student's turns and all observations are context. Only the teacher's visible reply is used. - 2059 training task variations, 32 episodes per step, 64 steps per epoch, fresh student prefixes and teacher continuations every epoch; AdamW, constant lr 1e-5, bf16, thinking off, 512 tokens per turn.
- Continuation: initialised from the epoch-3 weights (step 192 of the original run) with a fresh AdamW state; this checkpoint is step 128 of the continuation.
All checkpoints of this study (ScienceWorld test, 200 task variations)
Base Qwen3-1.7B scores 0.12%. Every number is the mean ± std of 4 independent passes at temperature 0.4, 512 tokens per turn, at most 30 rounds, thinking off, no system turn; success = final score 100, Avg@1 = mean final score / 100.
| method | epoch | success rate (avg of 4 passes) | Avg@1 | share of turns that are the literal open/close … | model |
|---|---|---|---|---|---|
| SFT | 1 | not evaluated | — | — | qwen3-1.7b-sciworld-sft-gpt54mini-ep1 |
| SFT | 2 | 13.25% ± 2.56 | 0.1574 | 74.20% | qwen3-1.7b-sciworld-sft-gpt54mini-ep2 |
| SFT | 3 | 12.88% ± 1.24 | 0.1557 | 74.88% | qwen3-1.7b-sciworld-sft-gpt54mini-3ep |
| ROSE 10+5 | 1 | not evaluated | — | — | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep1 |
| ROSE 10+5 | 2 | 7.75% ± 1.92 | 0.1455 | 62.68% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep2 |
| ROSE 10+5 | 3 | 17.50% ± 2.29 (re-run: 15.38% ± 1.43) | 0.2942 (0.2512) | 35.22% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-3ep |
| ROSE 10+5 | 4 | 20.25% ± 2.25 | 0.3433 | 17.98% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep4 |
| ROSE 10+5 | 5 | 25.75% ± 1.30 | 0.3692 | 2.77% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep5 ← this model |
| ROSE 10+5 | 6 | 20.75% ± 1.82 | 0.2946 | 26.59% | qwen3-1.7b-sciworld-rose-s10t5-gpt54mini-ep6 |
How to read this table:
- The teacher (
gpt-5.4-mini, medium) copies the prompt's action templateopen/close OBJliterally in 29.89% of its own turns; the simulator rejects it every time. Students inherit the habit, and their test success tracks how strong the habit is in the particular checkpoint (last column). In online ROSE the habit oscillates during training, so single-checkpoint numbers are noisy: ROSE epoch 2 was saved at a relapse peak, epochs 3 and 5 near minima, epoch 6 during another relapse. - Re-evaluating the same checkpoint (ROSE epoch 3) with 4 fresh passes moved its success rate from 17.50% to 15.38%, so differences under about 3 points are within evaluation noise.
- SFT epochs 1–2 are mid-schedule checkpoints of one 3-epoch run with a cosine learning-rate schedule; only epoch 3 saw the full decay.
- ROSE epochs 4–6 continue from the epoch-3 weights with a freshly initialised AdamW (no optimizer state had been saved).
Prompt format
ReAct. AgentGym's ScienceWorld instruction as a user turn, a canned assistant acknowledgement, then one user turn per observation;
the model replies Thought:\n...\n\nAction:\n<one command>. Render with Qwen3's chat template and enable_thinking=False.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained(REPO); model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="bfloat16")
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt")