Qwen3.5-9B-OutsideTheBox-RL
This repository provides the WebShop environment-reward checkpoint developed for the paper Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?
Model Description
The model is initialized from Qwen3.5-9B-OutsideTheBox-SFT and further optimized through multi-turn interaction with the WebShop environment. Training uses task instructions paired with grounded good workflows and directly rewards the final raw WebShop outcome in [0, 1]; it does not train on bad-workflow prompts.
RL Data Selection
Candidate tasks come exclusively from successful trajectories in the WebShop TRAIN split. The SFT checkpoint is sampled eight times per task, and tasks with non-constant raw rewards are retained. This produces 153 selected tasks, deterministically divided into 139 training tasks and a disjoint 14-task probe set; official WebShop test tasks are not used for selection or training.
Training Configuration
| Setting | Value |
|---|---|
| Initial policy | Qwen3.5-9B-OutsideTheBox-SFT |
| Training | Multi-turn environment GRPO |
| Workflow condition | Grounded good workflow |
| Reward | Raw WebShop task outcome [0, 1] |
| Generations per prompt | 8 |
| Epochs | 1 |
| Learning rate | 1e-6 |
| KL coefficient | 0 |
| Loss | DAPO |
| Maximum tool calls | 10 |
| Maximum completion length | 4,096 tokens |
Intended Use
This checkpoint is intended for research on WebShop agents, outcome-based reinforcement learning, and selective reliance on external workflows. Reproducing the reported results requires the WebShop environment and the multi-turn interaction harness used in the paper.
Loading the Model
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ElvisWang111/Qwen3.5-9B-OutsideTheBox-RL"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)