ACT · SO-101 · 16 teleop demonstrations (baseline)

An ACT policy for the SO-ARM101 arm, trained on 16 teleoperated episodes. This is the shared base of the comparison: the other two checkpoints add 16 more episodes to this exact set, one adding teleoperation and one adding handheld demonstrations.

Published as the floor, not as a usable policy. It picks reliably in the centre of the workspace and collapses outside it.

The comparison

Three ACT policies share a training recipe exactly — 50,000 steps, batch 8, seed 1000, one wrist camera at 10 fps — and differ only in what they were trained on. Task: pick a small box from a 20-square grid and place it on a target.

† Measured on a separately trained arm of the same teleop episode count, not on this exact checkpoint. Only the middle row was scored on the checkpoint published here. A 12-attempt session on the 32-teleop condition scored 75%; both are on the record. Rates come from the operator's raw per-attempt records and are re-derived by verify_results.py.

Doubling teleoperation moved success from 17% to ~70%. Replacing that second half with handheld demonstrations reached 65% — about nine tenths of the gain, on 16% fewer training frames, from data collected with a $0 gripper mock-up and a GoPro instead of a second robot session.

A later 10-placement session on a rebuilt scene put the handheld arm at 60% (6/10), consistent with the 65%.

Do not rank these policies by action error

Held-out action error is in the table because it is cheap and reproducible, and because it is wrong here. Calibrated on the teleop pair, the 0.25° gap for the handheld arm predicts roughly 33% success. It measured 65%. A policy can track a demonstrator's joint trajectory closely and still fail to close the gripper, and the converse. Rank on the bench, not on this number.

Use it

pip install lerobot==0.6.1

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("robotfuel/act_so101_t16b")
policy.eval()

# observation.state: (6,) joint positions, degrees
# observation.images.wrist: (3, 480, 640) uint8, wrist camera
action = policy.select_action(observation)   # (6,) target joint positions

The policy expects a single wrist camera and 6 joint positions, and emits absolute joint targets at 10 Hz. It was evaluated with n_action_steps=30.

Training

Every checkpoint in this set uses these settings unchanged. The only variable is the dataset.

Evaluation protocol

Attempts are run on a printed 20-square grid. The box is placed by hand at a prescribed coordinate, the arm is homed and held under torque, and the episode runs 35 s to completion without intervention. Scene brightness is measured before each attempt and attempts outside the trained range are recorded as skipped rather than failed, so they leave the denominator instead of deflating the rate.

An attempt counts as a success if the box ends on the target, including a scrappy recovery. Attempts that were skipped, voided by a rig fault, or interrupted are not scored.

Limitations

One task, one arm, one operator, one training seed per condition. Sample sizes are 12–20 attempts, which separates 17% from 65% comfortably but does not separate 65% from 70%. The policy is sensitive to lighting outside the trained range and to box placements at the far corners of the grid. Nothing here has been tested on a second robot, a second task, or a second scene.

Where this comes from

This repo is one artifact of a single study: can cheap handheld demonstrations substitute for robot teleoperation? The three policies below differ only in training data, so they are only meaningful read together.

• Writeup, with the method, the charts and the failure cases — https://robotfuel.fly.dev/research/handheld-demonstrations.html
• Code, raw per-attempt records and the verifier that recomputes every published number — https://github.com/sarafmudit/robotfuel_research/tree/main/handheld-demonstrations

Policies

• robotfuel/act_so101_t16b — 16 teleop (baseline) ← you are here
• robotfuel/act_so101_t16b_u16 — 16 teleop + 16 handheld
• robotfuel/act_so101_t16b_t16 — 32 teleop (matched-count control)

Datasets

• robotfuel/so101_t16b — 16 teleop episodes
• robotfuel/so101_t16b_u16 — 16 teleop + 16 handheld episodes
• robotfuel/so101_t16b_t16 — 32 teleop episodes
• robotfuel/so101_retargeted_umi — handheld GoPro demos, retargeted to joint space
• robotfuel/so101_pick_place_50_20260917_v3 — 50-episode teleop pick-and-place set

Every success rate published anywhere in this set is re-derived from the raw records by the verifier script; no number is transcribed by hand.