physicalai-bmi/nano-vla-pixels
A policy trained on real MuJoCo pixels — frames captured live from the Institute's in-browser Forge arm sim, not a synthetic renderer. It watches the actual rendered scene and outputs the joint-delta action, distilling our state-based checkpoint (forge-arm-reach-bc) into a pixel policy on genuine sim renders.
- Input: 3 stacked 48×48 RGB frames of the real MuJoCo render (temporal context = velocity).
- Output: a 3-D joint-delta action.
- Params: 186,099.
How the data was made
The state checkpoint drove the real Forge arm in a headless browser (WebGPU/Metal);
2,500 frames across 128 reaches were captured by screenshotting the WebGPU canvas
(a page's own JS cannot read WebGPU pixels — only an external compositor screenshot
can), each paired with the policy's action. See v2/tools/nanovla/capture.cjs.
Results (held-out reaches, real pixels)
| metric | value |
|---|---|
| Held-out action MSE | 7.26 × 10⁻⁵ |
| Predict-the-mean baseline | 4.25 × 10⁻⁴ |
| Variance explained | 82.9% |
| Single-frame (no temporal context) | 46.6% |
The jump from 46.6% → 82.9% with 3-frame stacking shows the action is velocity-dependent — a single still frame under-determines it. Reading fine control from coarse real renders is genuinely hard (and needs far more data than a state policy), so this is a research artifact, not a production controller.
Honest limitation
It can be run offline on captured frames (there's an in-browser replay at
https://physicalai-bmi.org/research/vla), but a live in-page closed loop is
impractical today — and it's worth being precise about why. Reading the render
back into JS is the catch: the 2D/getImageData/screenshot path reads blank on a
WebGPU canvas (it's double-buffered, not preserved after present). WebGPU's proper
readback (copyTextureToBuffer + mapAsync) is real, and three.js exposes
readRenderTargetPixelsAsync, but that path is currently unreliable in three.js's
WebGPU backend when rendering alongside an animation loop (three.js #31658, #31654).
So an external capture is the dependable route today. It is impractical, not
fundamentally impossible.
Files: model.safetensors, vla.web.json (float32 for in-browser; forward verified
vs safetensors), metrics.json, demo/ (real held-out frame triplets). CC-BY-4.0.