smolvla-so101-pickplace — language-conditioned pick-and-place
Successor: smolvla-so101-pickplace-v2 reaches 90% on the same protocol (trained on the 160-episode
-v2recovery set, which fixed this model's right-side coverage failures).
SmolVLA (450M) fine-tuned on
ahmedsohail2003/so101-sim-pickplace —
100 language-labeled MuJoCo demonstrations of an SO-ARM100 arm performing:
"Pick up the red block and place it in the blue tray."
Results — measured, same protocol for every row
20 evaluation episodes in the MuJoCo work-cell (fixed eval seed disjoint from training data, nominal scene, 240-step cap at 15 Hz):
| Policy | Success | Notes |
|---|---|---|
| SmolVLA base, zero-shot | 0/20 (0%) | never engages the block |
| SmolVLA fine-tuned (this model) | 11/20 (55%) | consistent when it engages: every success ≈105 steps, tray placement within ±3 mm |
| ACT (~52M, specialist BC, same 100 demos) | 13/20 (65%) | from the companion sim2cell project |
| ACT + temporal ensembling | 15/20 (75%) |
Honest read: 12k steps of expert-only fine-tuning takes a generalist VLA from 0% to within 10 points of a specialist behavior-cloning baseline on the same data — while being commandable in natural language. Failures are non-engagements clustered on right-side block spawns (under-represented in the 100 demos), not failed grasps: the policy either commits and succeeds consistently, or never engages.
Fine-tuning recipe (free-tier: single Kaggle T4)
- LeRobot
0.6.0,lerobot-train, AMP fp16, batch 8, 12k steps (~8 h) - Frozen VLM backbone, action expert only (
freeze_vision_encoder=True,train_expert_only=True) → ~100M trainable of 450M total - lr 1e-4 peak, cosine decay, warmup 1k; loss 0.97 → ~0.06
- Camera mapping from the pretrained base's naming:
--rename_map='{"observation.images.front": "observation.images.camera1", "observation.images.wrist": "observation.images.camera2"}'(2 dataset cameras are a subset of the base's 3; the absent third is skipped)
Use
import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
repo = "ahmedsohail2003/smolvla-so101-pickplace"
policy = SmolVLAPolicy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(
policy_cfg=policy.config, pretrained_path=repo,
preprocessor_overrides={"device_processor": {"device": "cuda"}},
)
batch = {
"observation.state": state_6d, # (1, 6) float32
"observation.images.camera1": front_rgb, # (1, 3, 224, 224) in [0, 1]
"observation.images.camera2": wrist_rgb, # (1, 3, 224, 224) in [0, 1]
"task": "Pick up the red block and place it in the blue tray.",
}
action = post(policy.select_action(pre(batch))) # (1, 6) joint setpoints
Evaluation script: eval_smolvla.py
in the TalkToTheCell project (env + protocol from the sim2cell work-cell).
Provenance
- Base: lerobot/smolvla_base (Apache-2.0)
- Data: author-recorded scripted-expert demonstrations in MuJoCo (dataset card)
- Robot model: MuJoCo Menagerie
trs_so_arm100(Apache-2.0)