physicalai-bmi/nano-vla-stack
A multi-step manipulation vision-language-action policy that runs fully in your browser. Instruction: "stack the red on the green." The arm grasps the named source block, carries it, and places it on the target block — a real two-phase task, not a single reach. Run it live at https://physicalai-bmi.org/research/vla.
- Vision: a 32×32 render of a 2-link arm + a red and a green block. When a block is grasped it is drawn at the gripper, so the phase (reaching vs carrying) is visible to the policy.
- Language: the source block is parsed from the instruction (first colour mentioned = source; open-vocab colour grounding) → a 2-way source one-hot. This is deliberate: mean-pooled sentence embeddings are order-blind ("red on green" vs "green on red" classifies at ~0.67), so we parse the order and let the policy condition on the resolved source/target.
- Action: two joint deltas. Grasp/place are automatic on proximity; the policy drives the arm.
- Params: 55,666.
Results (held-out, closed-loop, full episode)
| metric | value |
|---|---|
| Stack success (grasp → carry → place the correct block) | 94.0% |
| Grasp rate | 94.2% |
| Success with the source instruction flipped | 22.8% |
The flipped row shows the language matters: tell it to move the other block and it does, so success measured against the original target drops sharply. Trained by behavior cloning on a Jacobian-IK two-phase expert (100% ceiling) with DART state-noise injection to survive closed-loop drift across both phases.
Files: model.safetensors, vla.web.json (float32 for in-browser; forward verified
7.6e-8 vs safetensors), metrics.json. CC-BY-4.0, Institute for Physical AI @ BMI.
See also nano-vla-arm-3d (perspective reach) and nano-vla-reach.