act-so101-pickplace-v2 — a documented negative result
ACT (~52M) trained on
so101-sim-pickplace-v2
(160 SO-ARM100 pick-and-place episodes, 12.5% of which contain deliberate
first-grasp perturbations followed by verified expert recovery). The hypothesis:
the nominal baseline's failures were all grasp-miss-then-empty-transport, so
adding recovery demonstrations should teach the policy to retry.
It didn't work — and the way it failed is the useful part.
Results (same eval protocol as every model in this series)
| Policy | plain | + temporal ensembling | robustness n/v/p/f (15 eps each) |
|---|---|---|---|
| ACT v1 (100 nominal eps) | 65% | 75% (+10) | 60 / 60 / 53 / 40 |
| ACT-v2 (this model) | 50% | 35% (−15) | 47 / 33 / 40 / 33 |
| ACT-DR (150 domain-randomized eps) | 90% | — | 87 / 87 / 87 / 100 |
The ensembling sign flip is the diagnostic. Temporal ensembling averages overlapping action chunks: on v1's behaviorally unimodal data it smooths noise (+10); on recovery data — where near-identical states continue either "proceed to lift" or "re-approach" — it averages two incompatible modes into compromise actions (−15). Several v2 successes run ~40% longer than the stereotyped episode, consistent with the retry skill partially transferring, but the loss in grasp precision dominates.
The data-centric lesson, paired with the companion ACT-DR result: diversity that multiplies visual/physics variation while keeping behavior unimodal (domain randomization) improved everything (+25 nominal, flat-robust); diversity that multiplies behavioral modes (recovery branches) hurt this chunked-BC architecture. Handling multimodal continuations is exactly what diffusion/flow policies and VLA action experts are for.
Confound, noted honestly: at fixed 60k steps, v2 sees 24 epochs of 160 episodes vs v1's 43 epochs of 100; per-episode exposure is lower. The consistent direction across plain, ensembled, and randomized evals makes under-training an unlikely full explanation.
Training
LeRobot 0.6.0 lerobot-train, single Kaggle T4, identical recipe to v1/DR:
--policy.type=act --policy.chunk_size=45 --policy.n_action_steps=15 --batch_size=8 --steps=60000
Provenance
Data recorded by a verify-and-retry scripted expert (success-filtered) in the
author's MuJoCo work-cell; robot model from
MuJoCo Menagerie
(trs_so_arm100, Apache-2.0).