cup_pnp — yellow-cup pick-and-place policies
Four behavior-cloning policies for the task "pick up a yellow cup and place it on a baking tray," trained from egocentric human video reconstructed into robot demonstrations. This is a 2×2 ablation:
| architecture × observation | current (raw tray) | sim-tray |
|---|---|---|
π0.5 (LoRA on pi05_base) | pi05_current_tray/ | pi05_sim_tray/ |
| ABC-DiT (from scratch) | abc_dit_current_tray.pt | abc_dit_sim_tray.pt |
- Action / state space: 28-D — right arm only (3-slide + 3-hinge wrist + 22 finger joints), absolute joint targets.
- Observation: single rendered robot-view RGB (224×224).
- sim-tray variant re-renders the tray from its simulation mesh (inpainted over the original footage); the current variant keeps the tray as raw footage. Actions are identical across the two variants — only the observation pixels differ.
Results — held-out action-prediction MSE (normalized, 10-step horizon; lower is better)
| policy | current (raw tray) | sim-tray |
|---|---|---|
| π0.5 | 0.021 | 0.021 |
| ABC-DiT | 0.345 | 0.318 |
π0.5 (pretrained VLA + LoRA) is ~16× more accurate than ABC-DiT (trained from scratch with a randomly-initialized visual backbone). The sim-tray processing is neutral for π0.5 and a small improvement for ABC-DiT.
Contents
pi05_current_tray/,pi05_sim_tray/— π0.5 checkpoints (orbaxparams/+assets/norm-stats). Load with openpipolicy_config.create_trained_policy. Config:pi05_cup_v1/pi05_cup_v2.abc_dit_current_tray.pt,abc_dit_sim_tray.pt— ABC-DiT checkpoints ({model, step, norm_stats}+ optimizer). Load with ABC. state/action dim = 28, camera-keys =top.
Trained on 80 verified reconstructed demonstrations. See the project report for data provenance and method details.