MolmoAct2 x SO-101 sim, LoRA champion (v6)
MolmoAct2 is Ai2's open 5B vision-language-action model, shipped in LeRobot. This is a LoRA fine-tune of allenai/MolmoAct2-SO100_101 that picks up a red cube and places it on a blue target in a MuJoCo simulator of the SO-101 arm: from 0% zero-shot to 93% grasp in five days on one 24 GB GPU, with no human demonstrations.
Full story with videos: project page.
Results
Scored on 30 held-out cube positions, never seen in training:
| Metric | v6 |
|---|---|
| Grasp (cube held in the gripper) | 93% |
| Strict success (placed on target, arm at rest) | 9/30 (30%) |
| Loose (released within 6 cm of target) | 50% |
Recipe
- 500 scripted-expert demos in the sim, with injected noise and recovery (DART-style)
- Binary gripper (open/closed), success-state endings, slow release
- 10-step action chunks, absolute joint targets
- LoRA on the VLM plus full training of the action expert, one 24 GB GPU
The single biggest win was the binary gripper. Delta actions hurt, so v6 drops them.
Use it
from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy
pol = MolmoAct2Policy.from_pretrained("ataghof/molmoact2-so101nexus-lora-champion")
pol.config.inference_action_mode = "continuous" # not saved at train time
pol.to("cuda").eval()
The full eval harness (rollouts in the SO-101 sim, scoring, video) is in the repo.
Links
- Code, sim, collector, eval harness: https://github.com/ataghof/molmoact2-so101-sim
- Training dataset: https://huggingface.co/datasets/ataghof/so101nexus-cube500-binary
Built on MolmoAct2 (Ai2), LeRobot (Hugging Face), and so101-nexus (John Sutor).