Skill1 skill-only policy on WebShop (Qwen3.5-4B, step 40)
An RL checkpoint of Qwen3.5-4B trained with GRPO on WebShop under the Skill1 framework, in the skill-only ablation: the policy is updated only through the skill-selection steps (retrieval-query generation and skill re-ranking). The executor's environment actions are rolled out with the same model but are excluded from the training batch, so they never produce a gradient.
This checkpoint does not improve on the untrained base model. It is published as an ablation artifact and a reproducibility record, not as a better WebShop agent. See Results.
It is the WebShop counterpart of skill1-alfworld-skill-only-qwen3.5-4b-step40, run with the same code and the same ablation, but see "Differences from the ALFWorld run" below — they are not hyperparameter-identical.
Training summary
| Base model | Qwen/Qwen3.5-4B, chat-template thinking disabled (enable_thinking=False) |
| Algorithm | GRPO, group size 8, 16 tasks per step, lr 1e-6, KL loss coef 0.01 |
| Trainable phases | retrieval-query generation + skill re-rank. Executor play steps excluded from the training batch |
| Rewards | task outcome, discounted back into the selection steps; re-rank NDCG x 1.0; distillation intrinsic reward x 1.0 (verl defaults; the WebShop launch script passes no lambda overrides) |
| Skill library | top-k 3 retrieval, utility EMA alpha 0.05, admission requires a binary task success; 19 skills at step 40 |
| Episode length | env.max_steps=15, invalid-action penalty 0.5 |
| Steps | 40 (checkpoint every 10 steps), 10 h 01 m wall clock |
| Hardware | 4 x NVIDIA L40S (46 GB), vLLM 0.19.1 rollouts, FSDP actor with parameter/optimizer offload |
Results
64 WebShop goals per validation pass, temperature 0.4.
| step | success rate, with retrieval | success rate, no retrieval | reward (val/text/test_score) | skills in library |
|---|---|---|---|---|
| 0 (base) | 0.125 | 0.078 | 0.742 | 0 |
| 10 | 0.047 | 0.062 | 0.269 | 5 |
| 20 | 0.031 | 0.141 | 0.171 | 12 |
| 30 | 0.078 | 0.094 | 0.290 | 17 |
| 40 (this checkpoint) | 0.078 | 0.047 | 0.335 | 19 |
Read these numbers with care:
- The 64 goals are re-sampled at every validation pass from the held-out split (goal indices 0-499; training draws from 500-6909), so the rows are not scored on a common task set. A difference of one to five successes out of 64 is not meaningful.
- With-retrieval success is below the step-0 baseline at every evaluated step. The no-retrieval column moves too, even though the executor receives no gradient, which is consistent with sampling noise dominating at this scale.
val/text/test_scoreis the mean shaped reward, not a success rate. WebShop also reports a partial-credit metric whose key literally contains a space,webshop_task_score (not success_rate); it is not quoted here to avoid confusion.
Why the skill mechanism never engaged
Two measurements from the run explain the flat/declining with-retrieval numbers:
- Re-ranking produced no signal at all.
bigen_train/rerank_parse/successwas0out of 128 attempts at every step, andrerank_ndcgwas0.000throughout training and validation. The policy answered<rank>none</rank>every time, so the NDCG reward term contributed nothing to the gradient. - The skill library stayed tiny. Distillation admits a skill only from a trajectory that actually succeeded (binary success), and WebShop binary success ran at 0-13% per step, so the library grew to only 19 entries in 40 steps. Retrieval therefore returned mostly irrelevant candidates.
In effect the two trainable phases were optimised almost entirely by the discounted task return, with no usable intrinsic signal. On ALFWorld, where binary success is far more common, the same code grew a 1865-entry library over the same number of steps.
Differences from the ALFWorld run
| ALFWorld checkpoint | this WebShop checkpoint | |
|---|---|---|
lambda_rerank / lambda_distill | 0.3 / 0.3 (explicit overrides) | 1.0 / 1.0 (verl defaults; no overrides passed) |
env.max_steps | 50 | 15 |
| invalid-action penalty | 0.1 | 0.5 |
| library at step 40 | 1865 skills | 19 skills |
Environment (important for comparability)
These numbers are not comparable to published WebShop results:
- The catalog is a locally cut 1000-product subset: the first 1000 entries of
items_shuffle.jsonwithitems_ins_v2.jsonfiltered to those ASINs, cut on 2026-09-16. It is not the WebShop authors' releaseditems_shuffle_1000.json. It is selected by the repo defaultenv.webshop.use_small=True, withhuman_goals=False. - Search is served by the environment's
rank_bm25BM25Okapi fallback, not by the pyserini/Lucene index the original WebShop uses, because pyserini is not installed in this environment.
Usage
Standard Qwen3_5ForConditionalGeneration; loads with transformers >= 5 or vLLM. Trained
with the chat template's thinking mode off:
from transformers import AutoTokenizer, AutoModelForImageTextToText
repo = "YuyangZhao/skill1-webshop-skill-only-qwen3.5-4b-step40"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "..."}]
prompt = tok.apply_chat_template(messages, tokenize=False,
add_generation_prompt=True, enable_thinking=False)
The WebShop prompts expect an action inside <action> </action>; the skill-selection prompts
expect <query>...</query> and <rank>...</rank>.
Notes and limitations
- Weights are bf16 in a single unsharded
model.safetensors. - The base model's 15
mtp.*tensors (multi-token-prediction head) are not included in the merged checkpoint, althoughconfig.jsonstill advertisesmtp_num_hidden_layers: 1. MTP speculative decoding is therefore unsupported here. generation_config.jsonis synthesised from the model config by the merge step, not inherited from the base repo (the local base snapshot ships none), so it carries library defaults rather than Qwen's recommended sampling settings.- The vision tower of the base model is carried through unchanged and unused.
- The skill library itself is not part of this repository.
- Training run:
wandb.ai/yuyangzhao-ustc/skill1_webshop/runs/49f1o5r3.