physicalai-bmi/nano-vla-reach
A tiny open-vocabulary vision-language-action (VLA) policy that runs fully in your browser — no server, no GPU, no install. It looks at a scene, reads a free-form instruction you type, and acts. Run it live at https://physicalai-bmi.org/research/vla.
A VLA maps what it sees + what it's told → what it does. Frontier VLAs (π0, GR00T, SmolVLA) are hundreds of millions of parameters and need a GPU. This policy is 63,026 parameters; its whole forward pass — convolutions included — runs in plain JavaScript. The language is open-vocabulary: instructions are embedded on-device by
all-MiniLM-L6-v2(the same encoder at train and inference), so paraphrases it never saw still work. A faithful miniature of the paradigm, meant to be understood and run.
What it does
- Vision: a 24×24×3 image of a scene with a red target and a green target at random positions, plus the end-effector.
- Language (open-vocab): any instruction → a 384-d mean-pooled, normalized embedding from
all-MiniLM-L6-v2. "reach the red target", "grab the green one", "head toward crimson", "I want green" all work. - Action: a 2-D step for the end-effector.
The task is built so the policy must use both modalities: targets move every scene (it has to see them) and either can be the goal (it has to read the instruction).
Architecture
Conv(3→8, s2) → Conv(8→16, s2) → flatten(576) → concat[ text embedding(384) ] → MLP(→64→2), SiLU. 63,026 parameters. The 384-d language vector comes from all-MiniLM-L6-v2, run identically in Node (training) and transformers.js (browser) for exact parity.
Results (honest, closed-loop, held-out scenes)
| metric | value |
|---|---|
| Reaches the correct target — training instructions | 98.8% |
| Reaches the correct target — novel instructions it never trained on | 97.5% |
| Correct-target success when the instruction is flipped | 0.0% |
The novel-instruction row is the open-vocab result: unseen phrasings still reach the right target. The flipped row proves it truly obeys language — with the wrong instruction it goes to the other target. Small reference policy on a synthetic task, not a foundation model, not safety-tested for deployment.
Files
model.safetensors— weightsvla.web.json— the same weights as float32 arrays for in-browser inference (conv+MLP, verified bit-for-bit vs safetensors; max abs diff 7×10⁻⁸)metrics.json,inference.py
Language embedding (must match at train + inference)
from sentence_transformers import SentenceTransformer
enc = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
lang = enc.encode("grab the green one", normalize_embeddings=True) # (384,)
# then act(img, lang) — see inference.py
In the browser we use transformers.js Xenova/all-MiniLM-L6-v2 (pooling:'mean', normalize:true, dtype q8).
Released under CC-BY-4.0 by the Institute for Physical AI @ BMI. See also our first
released checkpoint, forge-arm-reach-bc.