act-tshirt-folding-lerobot
ACT (Action Chunking Transformer) policy for bimanual t-shirt folding on the Trossen AI stationary (WidowX AI) platform. Trained with lerobot. This Run Record documents the step-100000 (final) checkpoint.
Demo — on-robot eval
Prompt: Fold the tshirt
Offline on-robot evaluation of this checkpoint (step-100000) on the Trossen AI stationary bimanual platform — t-shirt folding.
TL;DR
- Policy: ACT (Action Chunking Transformer) — ResNet18 vision backbone + transformer encoder/decoder with CVAE.
- Framework: lerobot-latest (
--policy.type=act). - Method: Full fine-tune (no LoRA/adapters) from scratch on the target dataset.
- Robot: Trossen AI stationary bimanual, 14-DoF, 3 cameras, 30 fps.
- Checkpoint: step 100000.
Setup / rig configuration
Recorded on the Trossen AI Mobile kit with the mobile base disabled — arms mounted in a fixed, stationary configuration (functionally equivalent to the stationary kit for this task).
| Dimension | Value |
|---|---|
| Arm base height off ground | 39.5 in |
| Arm base-to-base spacing | 95 in |
| Arm horizontal offset from table edge | 8.25 in |
Model configuration (config.json)
| Field | Value |
|---|---|
| type | act |
| input_features | observation.state (14); images cam_high, cam_left_wrist, cam_right_wrist (3×480×640) |
| output_features | action (14) |
| chunk_size / n_action_steps | 100 / 100 |
| n_obs_steps | 1 |
| normalization | STATE=MEAN_STD, ACTION=MEAN_STD, VISUAL=MEAN_STD |
| vision_backbone | resnet18 (ResNet18_Weights.IMAGENET1K_V1) |
| dim_model | 512 |
| n_heads | 8 |
| dim_feedforward | 3200 |
| n_encoder_layers / n_decoder_layers | 4 / 1 |
| use_vae | true |
| latent_dim | 32 |
| n_vae_encoder_layers | 4 |
| dropout | 0.1 |
| kl_weight | 10.0 |
Training hyperparameters
| Field | Value |
|---|---|
| optimizer | AdamW |
| learning rate | 1e-05 |
| lr_backbone | 1e-05 |
| betas | (0.9, 0.999) |
| eps | 1e-08 |
| weight_decay | 0.0001 |
| grad_clip_norm | 10.0 |
| batch_size | 8 |
| total steps | 100,000 |
| save_freq | 5,000 |
Inputs (reproducibility manifest)
- Dataset:
TrossenRoboticsCommunity/mobile-baseoff-tshirt-folding-lerobot-v1. - Framework: lerobot-latest (
--policy.type=act). - Batch size: 8.
- Total steps: 100,000 (this is the final checkpoint).
- Training env / hardware: local RTX 5090 (TOTL workstation, Trossen cloud pipeline).
Outputs
- Weights:
model.safetensors(full fine-tune, no separate adapter). - Config:
config.json,train_config.json. - Processors:
policy_preprocessor.json(+ normalizer state),policy_postprocessor.json(+ unnormalizer state).
Evaluation (offline policy eval)
Verified to load and run directly via lerobot-record with --policy.path (no async policy server required for ACT). Run from the lerobot_trossen workspace:
uv run lerobot-record \
--robot.type=bi_widowxai_follower_robot \
--robot.left_arm_ip_address=192.168.1.4 \
--robot.right_arm_ip_address=192.168.1.5 \
--robot.id=bimanual_follower \
--robot.cameras='{
cam_high: {type: intelrealsense, serial_number_or_name: "230322276010", width: 640, height: 480, fps: 30},
cam_left_wrist: {type: intelrealsense, serial_number_or_name: "230322271348", width: 640, height: 480, fps: 30},
cam_right_wrist: {type: intelrealsense, serial_number_or_name: "128422270115", width: 640, height: 480, fps: 30}
}' \
--display_data=false \
--dataset.repo_id=hamza-trossen/eval_act_tshirt_folding \
--dataset.push_to_hub=false \
--dataset.num_episodes=20 \
--dataset.episode_time_s=105 \
--dataset.reset_time_s=25 \
--dataset.single_task="Fold the tshirt" \
--policy.path=TrossenRoboticsCommunity/act-tshirt-folding-lerobot \
--robot.min_time_to_move_multiplier=8.0
The task prompt used during recording and eval was "Fold the tshirt".
Eval Results
| Eval Task | Reps | Success | Notes |
|---|---|---|---|
| tshirt-folding | 10 | 6/10 (60%) | Policy confirmed working on hardware; min_time_to_move_multiplier=8.0 config performed best for ACT on this task |