Skill1 skill-only policy on ALFWorld (Qwen3.5-4B, step 40)
An RL checkpoint of Qwen3.5-4B trained with GRPO on ALFWorld under the Skill1 framework, in the skill-only ablation: the policy is updated only through the skill-selection steps (retrieval-query generation and skill re-ranking). The executor's environment actions are rolled out with the same model but are excluded from the gradient, so any improvement comes from skill selection / distillation and the growing skill library rather than from the executor learning the task.
Training summary
| Base model | Qwen/Qwen3.5-4B (thinking mode disabled via enable_thinking=False) |
| Algorithm | GRPO, group size 8, 16 tasks per step, lr 1e-6, KL coef 0.01 |
| Trainable phases | query generation + re-rank (executor play steps excluded from the batch) |
| Rewards | task outcome (discounted back to the selection steps), re-rank NDCG x 0.3, distillation intrinsic reward x 0.3 |
| Skill library | top-k 3 retrieval, utility EMA alpha 0.05, 1865 skills at step 40 |
| Steps | 40 (checkpoint saved every 10 steps) |
| Hardware | 4 x NVIDIA L40S, vLLM 0.19.1 rollouts, FSDP actor |
Validation (64 held-out ALFWorld tasks, temperature 0.4)
| step | success rate with retrieval | success rate without retrieval |
|---|---|---|
| 0 (base) | 0.281 | 0.406 |
| 20 | 0.344 | 0.391 |
| 30 | 0.484 | 0.438 |
| 40 (this checkpoint) | 0.359 | 0.391 |
The executor's own success rate (without retrieval) stays flat, as intended for the ablation; the with-retrieval number depends on the skill library used at inference.
Usage
The checkpoint is a standard Qwen3_5ForConditionalGeneration model and loads with
transformers >= 5 or vLLM. It was trained with the chat template's thinking mode
turned off:
from transformers import AutoTokenizer, AutoModelForImageTextToText
repo = "YuyangZhao/skill1-alfworld-skill-only-qwen3.5-4b-step40"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "..."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
The ALFWorld prompts expect the model to reason inside <think> </think> and then
emit one admissible action inside <action> </action>; the skill-selection prompts
expect <query>...</query> and <rank>...</rank> outputs.
Notes
- Weights are bf16 in a single
model.safetensors; the vision tower of the base model is included unchanged (it is not used for ALFWorld). - The skill library itself is not part of this repository.