so101_pick_place_pen_smolvla
A SmolVLA vision-language-action policy fine-tuned with LeRobot for a pick-and-place pen task on the SO-101 robot arm.
이 모델은 SO-101 로봇 팔에서 펜 집어 옮기기(pick-and-place) 작업을 수행하도록 LeRobot으로 파인튜닝한 SmolVLA 비전-언어-액션 정책입니다.
Model Details / 모델 정보
- Policy / 정책: SmolVLA (fine-tuned from
lerobot/smolvla_base) - Robot / 로봇: SO-101 follower arm
- Task / 작업: Pick and place a pen / 펜 집어 옮기기
- Dataset / 데이터셋:
nota-gmbh/pick_and_place_pen_so101 - Episodes / 에피소드: 53
Training / 학습
- Steps: 40,000
- Hardware: Multi-GPU
- Batch size: 64
- Base model:
lerobot/smolvla_base
During data collection, the camera frames were captured keeping a consistent width-to-height aspect ratio across episodes for visual consistency.
데이터 수집 시, 에피소드 전반에 걸쳐 카메라 프레임의 가로:세로 비율을 동일하게 유지하려고 노력했습니다.
export NCCL_P2P_DISABLE="1"
export NCCL_IB_DISABLE="1"
export HF_USER="~~"
export TASK_NAME="pick_and_place"
accelerate launch --multi_gpu --num_processes=4 \
$(which lerobot-train) \
--policy.path=lerobot/smolvla_base \
--dataset.repo_id=Dangni/pick_and_place_so101_pen \
--dataset.video_backend=pyav \
--dataset.image_transforms.enable=true \
--dataset.image_transforms.max_num_transforms=3 \
--dataset.image_transforms.random_order=true \
--batch_size=16 \
--num_workers=1 \
--steps=100000 \
--output_dir=outputs/train/so101/smolvla/pick_and_place_$(date +%Y%m%d_%H%M%S) \
--job_name=smolvla_yellow \
--policy.device=cuda \
--policy.freeze_vision_encoder=true \
--policy.train_expert_only=true \
--policy.push_to_hub=false\
--policy.repo_id=Dangni/pick_and_place_yellow_smolvla \
--rename_map='{"observation.images.fixed": "observation.images.camera1", "observation.images.handy": "observation.images.camera2"}' \
--save_freq=5000 \
--wandb.enable=true
Optimizer / 옵티마이저
| Item / 항목 | Value / 값 | Note / 설명 |
|---|---|---|
| Optimizer | AdamW | decoupled_weight_decay=True |
| Initial LR | 1.0e-04 | Learning rate at the start of training / 학습 시작 LR |
| LR (latest logged) | 3.2594e-05 | LR at the logged step (scheduled decay) / 스케줄러 적용 후 LR |
| Betas | (0.9, 0.95) | 1st / 2nd moment coefficients |
| Epsilon | 1e-08 | Numerical stability constant |
| Weight Decay | 1e-10 | Very small L2 regularization |
Note: The learning rate decays over training via the scheduler, so
3.2594e-05reflects a logged intermediate step rather than the final value. 참고: LR은 스케줄러에 의해 학습 중 감소하며,3.2594e-05는 로깅된 중간 스텝의 값입니다.
Usage / 사용법
lerobot-rollout `
--strategy.type=base `
--policy.path=Dangni/pick_and_place_smolvla_so101_040000 `
--policy.n_action_steps=50 `
--robot.type=so101_follower `
--robot.port=COM3 `
--robot.id=follower_arm `
--robot.cameras='{fixed: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, handy: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}' `
--rename_map='{"observation.images.fixed": "observation.images.camera1", "observation.images.handy": "observation.images.camera2"}' `
--task="Pick the pen and place it inside the yellow border" `
--duration=500
Limitations / 한계
- Trained on a single pick-and-place pen task with 53 episodes; performance is limited to this task and the data-collection setup.
- Camera resolution and viewpoint should match the training setup for reliable inference.
- 단일 펜 집기 작업, 53개 에피소드로 학습되어 해당 작업과 수집 환경에 한정됩니다.
- 추론 시 카메라 해상도·시점을 학습 환경과 동일하게 맞춰야 안정적으로 동작합니다.
Links / 링크
- Base model:
lerobot/smolvla_base - Dataset:
nota-gmbh/pick_and_place_pen_so101 - LeRobot: https://github.com/huggingface/lerobot aa