CCPO return-hard + task backoff + J-weighting — Qwen2.5-7B-Instruct on ALFWorld
This is an interim checkpoint from a run that has not finished. Training targets 150 steps; this arm is at step 113. Numbers below will change. See Status at the bottom.
What this is
An agent policy trained with CCPO (Context-Conditioned Policy Optimization) on ALFWorld, a text-based embodied household benchmark. The base model is Qwen2.5-7B-Instruct.
CCPO targets credit assignment in multi-turn agent RL. Methods in the GRPO family split the advantage into an episode term and a step term. CCPO keeps the episode term untouched and replaces only the step term with a context-conditioned, uncertainty-shrunk leave-one-out baseline, so a head-to-head against the baseline isolates the estimator and nothing else.
The step term works like this. Entries are bucketed by (task_uid, observation_text),
the same anchor-state grouping GiGPO and G2PO use. Each entry gets a frozen feature
vector from the reference policy's last-prompt-token hidden state, whitened across
the batch. Siblings are weighted by exp(-d/tau) on that feature distance, and the
baseline is a weighted leave-one-out mean over other trajectories. A shrinkage factor
then pulls the result back toward the uniform baseline wherever the disagreement between
the two does not exceed its own sampling noise.
Code: tracyhann/agent-context-grpo, built on verl-agent.
This arm
The hard-gate arm plus two additions. backoff_task=1 gives rows whose anchor bucket holds no sibling a task-level baseline instead of zero credit, and jweight_c=1.0 scales the step term by J/(J+c) so small buckets count for less.
Flags that distinguish it: gate=hard, target=return, rho=0.0, backoff_task=1, jweight_c=1.0
Results so far
Held-out evaluation on 128 unseen ALFWorld tasks (eval_in_distribution), sampled at
temperature 0.4, which is the protocol the published baselines use.
Converged window (step >= 70), the number to read:
| metric | value |
|---|---|
| held-out success | 90.0% |
| standard deviation | 3.1 |
| evaluations averaged | 9 |
Best single evaluation, step 105:
| Held-out | Turns | Pick | Look | Clean | Heat | Cool | Pick2 |
|---|---|---|---|---|---|---|---|
| 96.1 | 10.8 | 100.0 | 100.0 | 100.0 | 96.2 | 91.7 | 89.4 |
Training success rate at that step was 89.1%. "Turns" is the mean number of environment interactions per held-out episode; it starts near 39 and falls as the policy learns to solve tasks directly rather than by searching.
Do not read the best row as the result. It is the maximum of a noisy series and is biased upward by roughly 1.5 standard deviations. A single 128-episode evaluation here carries about +/-5 points. The converged mean is the honest summary.
The three arms, side by side
Converged-window means, all on identical hardware, seed and data order:
| arm | held-out mean | sd | n |
|---|---|---|---|
| CCPO return-hard + task backoff + J-weighting | 90.0 | 3.1 | 9 |
| CCPO return-hard | 83.5 | 6.5 | 10 |
| CCPO global gate, successor-node target | 77.5 | 6.3 | 9 |
What this does and does not show
It does not show that CCPO beats GRPO. No valid GRPO baseline exists in this codebase; an earlier attempt was contaminated when a relaunch passed the run tag but not the estimator. No claim of superiority over GRPO is made here.
These numbers are not comparable to the published ALFWorld results. Those (GRPO 72.8, GiGPO K=2 90.16, HGPO K=2 92.77) are all Qwen2.5-1.5B-Instruct. This is a 7B model, so it is a different setting entirely, and a higher number here says nothing about the estimator relative to those methods.
Every arm here is one seed. The published rows are three. The repository's own guidance is to report the mean of at least three seeds and never to read single points, so differences between the arms above should be treated as unresolved until replicated.
What the comparison between arms does isolate is the estimator: the three runs share hardware, seed, batch and data order, and were verified to produce identical rollouts at step 1 before any gradient was applied.
Training setup
| base model | Qwen2.5-7B-Instruct |
| benchmark | ALFWorld (AlfredTWEnv), 6 task types |
| batch | 16 tasks x 8 rollouts = 128 episodes/step |
| max env steps | 50 per episode, history length 2 |
| learning rate | 1e-6, constant |
| KL | 0.01, low-variance estimator |
| gamma | 0.95 |
| max prompt / response | 2048 / 512 tokens |
| eval | 128 held-out tasks every 5 steps, T=0.4 |
| hardware | 8x NVIDIA H200 |
Hyperparameters follow the G2PO reference ALFWorld script so the protocol matches the published baselines; only the backbone size and GPU count differ.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SingleBicycle/ccpo-alfworld-qwen2.5-7b-return-hard-fbjw"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
The policy expects the ALFWorld agent prompt used by verl-agent and emits
<think>...</think><action>...</action>. The text inside <action> is passed to the
environment, so it must match an admissible action.
Status
Interim. This arm is at step 113 of 150. The checkpoint published here is the best-scoring one so far (step 105). Final weights and settled numbers will replace these once the run completes.
Citation
Built on verl-agent (Apache 2.0) and verl. ALFWorld is from alfworld/alfworld.