ACT · SO-101 · 8-bin pick-and-place · wrist + front · chunk 50
Action Chunking Transformer trained with LeRobot 0.6.0
on BrutalCaesar/phi_so101_8bin_v1.
Two cameras: wrist (gripper module) + front (desk-level, Logitech Brio 101).
One of six runs in a chunk-size × camera-pair sweep ({50, 75, 100} × {wrist+top, wrist+front}).
🚨 The camera keys are swapped — read this before inference
The dataset was recorded with the wrist and top camera keys transposed. This model therefore
expects the physical wrist camera under the key observation.images.top:
| Feed this physical camera | Under this observation key |
|---|---|
| wrist (gripper module) | observation.images.top |
| front (desk-level) | observation.images.front |
Wire the overhead camera to observation.images.top and the policy gets a close-up it has never
seen in that slot. It will not error — it will just behave badly. The key name is wrong;
the model is not.
Camera framing matters too: the mounts must sit where they sat during recording. A camera that drifted between recording and evaluation silently invalidates the policy.
Inputs / outputs
| Shape | Notes | |
|---|---|---|
observation.images.top | (3, 480, 640) | physical wrist camera, no resize anywhere in the pipeline |
observation.images.front | (3, 480, 640) | physical front camera |
observation.state | (6,) | SO-101 joint positions |
| action | (50, 6) | chunk of 50 future actions, all executed (n_action_steps=50) |
Normalization ships with the checkpoint as LeRobot 0.6.0 processor files
(policy_preprocessor*, policy_postprocessor*) — not baked into the policy. Load through
ACTPolicy.from_pretrained so they are picked up; hand-rolling a forward pass without them
produces actions in the wrong units.
Training
| Steps | 100,000 (= 800K samples, ≈14.6 epochs over 54,800 frames) |
| Batch size | 8 |
| Final train loss | 0.055 |
| Optimizer | AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4 |
| Backbone | ResNet-18, ImageNet-pretrained, not frozen |
dim_model / chunk_size / kl_weight | 512 / 50 / 10.0 |
n_obs_steps | 1 (single frame, no history) |
| Seed | 1000 |
| Hardware | 1× A100/H200, Northeastern Explorer cluster |
Everything except chunk_size and the camera pair is LeRobot default, deliberately — the
sweep is meant to be comparable across people training the same task on the same defaults.
Train / held-out split
Trained on 89 episodes covering 6 of the 8 bins. Left bin 3 and right bin 2 are held out entirely — no episode touching them appears in training. Those two bins are the generalization test; the other six measure fit.
Evaluation
Not yet evaluated on hardware. No success rate is reported here because none has been measured. The planned protocol is 20 scored rollouts per bin, reported per bin and never averaged — an average over 8 bins hides exactly the failure this split is designed to expose.
Train loss of 0.055 says the model fits the demonstrations. It says nothing about whether it completes the task.
Usage
from lerobot.policies.act.modeling_act import ACTPolicy
policy = ACTPolicy.from_pretrained("BrutalCaesar/act_so101_8bin_wrist_front_chunk50")
Or point a LeRobot eval script at it with --policy.path=BrutalCaesar/act_so101_8bin_wrist_front_chunk50,
remembering the camera-key mapping above when you write --robot.cameras.
Known limitations
- Single observation frame, so no velocity information; the policy infers motion from wrist-view context alone.
- The CVAE latent is effectively collapsed at
kl_weight=10.0on this dataset (KL ≈ 0 throughout training), so this behaves close to a deterministic chunk regressor. Not a defect — expected for ACT — but do not read the latent as a source of behavioral diversity. - Trained on one lighting setup in one room. No domain randomization beyond LeRobot's default image augmentation.