physicalai-bmi/nano-vla-arm-3d
An open-vocabulary vision-language-action policy that drives an articulated arm
from a perspective (3D-camera) view, running fully in your browser. Same
see-be-told-act task as nano-vla-arm,
but the scene is rendered through a fixed perspective homography — a receding
tabletop, foreshortened blocks — so the policy must read a camera-like image, not a
flat schematic. Run it live at https://physicalai-bmi.org/research/vla.
- Vision: a 40×40×3 perspective render of a 2-link arm + red/green blocks on a receding table.
- Language (open-vocab): any instruction, embedded on-device by all-MiniLM-L6-v2 (384-d).
- Action: two joint deltas. Whole forward pass (3 conv + MLP) runs in plain JS.
- Params: 119,986.
Results (held-out, closed-loop)
| metric | value |
|---|---|
| Reaches correct block, trained instructions | 97.0% |
| Reaches correct block, novel instructions | 97.0% |
| Correct when instruction flipped | 0.0% |
Trained by behavior cloning on a Jacobian-IK expert with DART state-noise injection.
The perspective view (baked 3×3 homography, stored in config.Hinv) is applied
identically in training and in the browser, so train == live. A small reference
policy on a rendered task; not a foundation model.
Files: model.safetensors, vla.web.json (float32 for in-browser, verified 2.3e-7
vs safetensors), metrics.json. CC-BY-4.0, Institute for Physical AI @ BMI.