SO-ARM101 pick-and-place — π0 full fine-tune (9 tasks)

A π0 flow-matching VLA policy fine-tuned on a 9-task SO-ARM101 tabletop pick-and-place dataset. Language-conditioned: one checkpoint serves all nine tasks, selected by the instruction string you pass at inference.

Trained with LeRobot (lerobot-train), stock recipe, full-parameter fine-tuning (vision encoder not frozen, action expert not trained in isolation).

The base-model tag may render as lerobot/pi0_old: upstream renamed that repo and lerobot/pi0 now redirects to it. It is the same weights this was fine-tuned from.

Tasks

Nine tabletop tasks. The instruction must be passed verbatim — the strings below are exactly the ones present in the training data, including the article oddity in a eraser and the missing it in place in the bin.

All nine were merged into a single training set with no re-balancing / no up-sampling — episode counts are already roughly uniform, so the merge is balanced as-is.

Data

• Source: zhuzhuangtian/so101-pick-place-tasks on ModelScope (Apache-2.0), LeRobot v3.0 format.
• Robot: SO-ARM101 follower arm, 6 DoF.
• Cameras: two RGB streams, observation.images.top and observation.images.wrist, 480×640 each, 30 fps.
• observation.state and action are both float32(6): shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper (joint positions, absolute targets — not deltas).

Training

Published checkpoint = step 25,000, not the final step. All 12 saved checkpoints were evaluated offline and step 25,000 was best on both metrics below; the curve is U-shaped (degrades after 25k), so training longer would have been worse.

Offline evaluation

There is no simulator for this rig, so evaluation is offline only, against a copy-the-current-joint-angles baseline:

• ratio = MAE(prediction, action) / MAE(current_state, action) — must be < 1, i.e. the policy must beat "just hold still". This is a much harder bar than raw correlation: the naive baseline already scores 0.93–0.997 correlation on every joint, so corr > 0.9 has no discriminative power on this data and is not used here.
• delta_corr = corr(prediction − state, action − state) — must be > 0 (direction of motion).
• delta_std — guards against a policy that collapses to shrinking outputs.

Published checkpoint (200 frames sampled uniformly across the whole dataset):

All six joints beat the baseline. Weakest axes are the two wrist DoF (wrist_roll, wrist_flex) — those are the most likely to misbehave first on hardware.

Real-robot status

One closed-loop run has been done: the policy drove a physical SO-ARM101 through Pick up a plush toy and place in the bin with both cameras live and inference served remotely from a GPU host, visually confirmed correct by a human observer. No quantified success rate exists. Treat this as "the checkpoint loads, the scale is right, and it moves sensibly", not as a validated success rate.

Usage

from lerobot.policies.pi0.modeling_pi0 import PI0Policy

policy = PI0Policy.from_pretrained("Harrysunshine/so101-pi0-9task")
# batch: observation.state (6,), observation.images.top (3,480,640),
#        observation.images.wrist (3,480,640), task = one instruction string from the table above
action = policy.select_action(batch)

Or as a LeRobot policy server for a low-power robot host:

# on the GPU host
lerobot-policy-server --host=0.0.0.0 --port=8080
# on the robot host
lerobot-robot-client \
  --policy_type=pi0 \
  --pretrained_name_or_path=Harrysunshine/so101-pi0-9task \
  --task="Pick up a plush toy and place in the bin" \
  --actions_per_chunk=50

The two *normalizer_processor.safetensors files are required. Without them LeRobot silently falls back to identity normalization: loss-free, but every action comes out at the wrong scale and the arm will fly off. Load the repo as a whole, do not cherry-pick model.safetensors.

Only inference assets are published here (weights, config, pre/post-processors, training config). Optimizer/scheduler/RNG state is not included, so this repo cannot be used to resume training.

Limitations

• Rig-specific top camera. The training data comes from a different physical setup than ours: the wrist view matches pixel-for-pixel (same SO-ARM101 follower hardware), but the top camera extrinsics differ — the same joint configuration images the arm at a different position in frame. Dropping this checkpoint onto another SO-ARM101 will not work without camera adaptation or a short fine-tune on data from the target rig.
• Offline metrics do not predict real-world success. Related work on other arms has produced checkpoints with 0.995 offline correlation that still failed to grasp on hardware.
• Absolute joint-position action space. Sending these as deltas, or recording target data with a per-session origin offset baked in, will diverge in closed loop.
• Fixed 480×640 dual-camera observation. Single-camera or differently-cropped inputs are out of distribution.
• Tabletop objects and bin from the training scene only; no generalization claims beyond them.

Related

• Harrysunshine/so101-smolvla-9task — SmolVLA on the same merged dataset, same evaluation harness (cost/accuracy comparison point).
• Harrysunshine/so101-act-cube — ACT on the single cube task.

License

Apache-2.0, following the lerobot/pi0 base model and the source dataset.

中文说明

SO-ARM101 桌面抓放 π0 全参微调权重，9 个任务合并训练，语言条件——一个 checkpoint 覆盖全部九个任务， 靠推理时传入的指令字符串区分。指令必须与上表逐字一致（注意 a eraser 和 place in the bin 里没有 it）。

• 数据：ModelScope zhuzhuangtian/so101-pick-place-tasks（Apache-2.0，LeRobot v3.0）， 2200 集 / 784,963 帧 / 双相机 480×640 / 6 维绝对关节位置。九任务全量合并，不做偏斜上采样。
• 训练：lerobot-train 官方入口，全参微调，30k 步，有效 batch 64，bf16，余弦退火 + 2000 步 warmup。
• 发布的是第 25,000 步，不是末点：12 个存点逐点离线验收，25k 双指标同时最优，曲线 U 形（25k 后退化）。
• 验收判据不用 corr > 0.9：「照抄当前关节角」这个恒等基线的逐关节 corr 就有 0.93–0.997，全部过门、零筛选力。 改用 ratio = MAE(预测) / MAE(照抄) 必须 < 1，配 delta_corr 与防坍缩的 delta_std。
• 六关节 ratio 全部 < 1；短板是腕部两个自由度（wrist_roll 0.906、wrist_flex 0.775）。
• 真机只做过一次闭环目检通过（抓毛绒玩具放进筐，双相机 + 远程 GPU 推理），没有量化成功率。
• 两个 *normalizer_processor.safetensors 必须一起加载，缺了会静默退回恒等归一化，动作尺度整体错、机械臂跑飞。
• 边界：这批数据的 top 相机外参与我们自己的臂不同（wrist 视图逐像素同款，top 不同）， 直接换机器用不了，需要相机适配或用目标机器的少量数据再微调。