State-value models V(s) for mimicgen coffee_d0
Two state-only value functions predicting the discounted return
(gamma = 0.99) from the 66-d proprioceptive state. Trained without any access
to a learned Q critic — only success/failure trajectories and the datasets' own
human annotations. The armF2 critic is used as an evaluation reference only.
v_td | v_relabel_hinge | |
|---|---|---|
| pearson vs armF2 Q | 0.854 | 0.888 |
| spearman vs Q | 0.870 | 0.857 |
| pearson vs Q, failure chunks | 0.671 | 0.725 |
| pearson vs Q, ±40 frames of the annotated failure | 0.609 | 0.632 |
| MAE vs Q | 0.1077 | 0.0772 |
| pearson vs true return G | 0.839 | 0.812 |
| MAE vs G | 0.0827 | 0.0882 |
Split B holdout (episodes 80-99 of each rollout set), stride-8 chunk grid, n = 1577. Three seeds each; the table scores the mean of the three predictions.
The two models
v_td — plain TD regression. Target
y = r + (1-done) * gamma^8 * V_target(s_{t+8}), smooth-L1, hard target sync
every 200 steps. Reward is 0 everywhere except the terminal frame of a success
episode, so once the +8 lookahead passes the end the target is the exact
closed-form return.
v_relabel_hinge — the same, plus two ingredients built from the annotations:
- Hindsight-relabelled failure targets. On a failure episode the ordinary
target is 0 at every frame and says nothing about when it went wrong.
Using
meta/lerobot_annotations.json, fort < t_failthe target becomes the matched success stage's return at the same stage fractionu = t / t_fail, and 0 after. Blendedy = 0.75 * y_TD + 0.25 * y_relabel. Mode -> stage:failed to grasp pod -> good grasp,pod misaligned -> good insertion. - A margin hinge on matched pairs.
relu(0.1 - (V(s_success) - V(s_failure)))over 5851 matched (success stage-end, failure-mode) state pairs, anchors jittered +-5 frames. A margin matters: the unbounded Bradley-Terry formsoftplus(-(V_succ - V_fail))blows the value range from ~1.2 to 5.2 and drops pearson-vs-Q to 0.438.
Usage
import torch, numpy as np
from modeling_v import VNet, scale_state
net = VNet(66)
net.load_state_dict(torch.load("weights/v_relabel_hinge_seed0.pt", map_location="cpu"))
net.eval()
x = torch.from_numpy(scale_state(raw_state_66d)) # scaling is REQUIRED
with torch.no_grad():
value = net(x)
scale_state applies the armF2 critic's own q01/q99 normalisation (shipped as
state_scaler.json) and clips to +-5. Feeding raw states will not work.
For the reported numbers, average the three seeds' predictions.
Known limitations
- Not usable for advantage.
gamma^8 * V(s') - V(s)correlates -0.111 with the true advantageQ - E_a[Q]and its sign agrees 45.2% of the time. The true advantage has sd 0.0047 against this quantity's 0.055 — at the prior's action spread the action explains 0.15% of Var[Q]. Use V as a baseline or a shaping potential, not as a source of action rankings. - Fit on prior-policy rollouts only (
mtdit_flow_60k). Expect degradation as a policy moves off that distribution. - The relabelling needs annotated failure frames, which fresh rollouts do not have. Retraining online requires predicted failure anchors.
p_headin the checkpoints is untrained; ignore it.
Data
chomeed/mimicgen_coffee_d0_224x224_mtdit_flow_60k_{success,failure}
(100 episodes each, 20 fps, 66-d state), including the
meta/lerobot_annotations.json markers the relabelling depends on.