SmolVLA — language-conditioned two-disc pick and place (SO-101, MuJoCo)
A SmolVLA policy fine-tuned to pick up a cube and place it on the disc named in a natural-language instruction. Two coloured target discs are visible at all times, so the picture alone cannot say which one is correct — the instruction is the only signal that identifies the goal.
This is the reported final model from the project: the checkpoint at 60,000 steps,
about two passes over the dataset. zwan1003/pickplace_skills_vla_v3_2 holds the same run
at 90,000 steps, which overfits and scores lower; it is kept for comparison.
Code: the environment, scripted expert controller, data collection pipeline and evaluation
harness are at github.com/zwaneiz/so101-vla-pickplace.
You need it to reproduce the evaluation, since the two-disc scene is a custom subclass of the
so101-nexus environment.
What it does
| Instruction | Behaviour |
|---|---|
pick up the cube | Grasp and lift the cube |
place it on the left / place it on the right | From a held cube, carry and set down on the named disc |
pick up the cube then place it on the left | The whole sequence |
Use these strings exactly. The policy was trained on them and is sensitive to wording (see Limitations).
Results
Measured on starting positions the model never trained on, averaged over three seeds.
| Measure | Result | Episodes |
|---|---|---|
| Grasp the cube | 33% | 150 |
| Place it, once held | 97% | 30 |
| Whole task | 37% | 120 |
| Went to the instructed disc | 96–100% | — |
| Went to the wrong disc | 2 of 120 | — |
The whole-task score is close to grasp × place. Placement is close to solved; grasping is the bottleneck, caused by a gripper tolerance of under 4 mm on a 25 mm cube.
It follows the words, not the colours. Recolour both discs grey and it still scores 87%. Swap the two colours so they contradict the instruction and it scores 100%, going to the instructed position every time. Remove the direction word and it drops to 3%.
Requirements
python 3.12
lerobot 0.4.4
mujoco 3.9.0
so101-nexus 0.3.12
The environment is a two-disc variant of the so101-nexus pick-and-place scene: a second
target disc is injected into the model so both a left and a right target are always present.
That subclass, and everything else needed to reproduce the evaluation, is in the
code repository.
Loading
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
policy = SmolVLAPolicy.from_pretrained("zwan1003/pickplace_skills_vla_v3_2_ckpt060k")
Four things that will silently break it
1. Camera names must be remapped. SmolVLA expects camera1/camera2; the dataset uses
descriptive names. Apply the same mapping used in training:
observation.images.overhead_camera -> observation.images.camera1
observation.images.wrist_camera -> observation.images.camera2
Get this wrong and the policy receives the wrist image where it expects the overhead view. It does not raise an error, it just behaves badly.
2. Match the physics. The training data was collected with a stiffened contact model and tangential friction scaled by 3.0, representing rubber fingertip pads. Evaluate with the same settings or the grasp will not transfer.
3. Seed torch per episode. SmolVLA samples noise at every inference through its flow-matching action head. Seeding the simulator alone is not enough; without seeding torch, the same command produces different results.
4. Normalisation comes from the dataset. State is normalised on input and actions are
de-normalised on output using the statistics of
zwan1003/pickplace_skills_v3_2.
Omitting this produces smooth but completely misdirected motion.
Limitations
- Narrow language. Near-synonyms are free:
grab the cubeandpick up the blockscore the same as the trained wording. Distant phrasing is not:carry the objectdrops to 7%, colour naming (place it on the blue disc) and negation both fail outright. - Grasping is the ceiling. The gripper needs better than 4 mm of accuracy. Raising contact friction further lifts whole-task success to about 56% without retraining, which shows the limit is mechanical rather than learned.
- Simulation only. No transfer to a physical arm has been tested or claimed.
- Restricted workspace. Cube positions are drawn from a measured reachable region, not the arm's full range.
Context
Produced for a final-year project investigating whether a compact vision-language-action model, fine-tuned on a small dataset using consumer hardware, genuinely conditions its behaviour on language rather than on visual shortcuts. The whole pipeline — data collection, training and evaluation — ran on a single laptop with a 4 GB GPU.