physicalai-bmi/nano-vla-pixels-dart
A pixel-only control policy for an in-browser MuJoCo arm, trained on frames captured through the same in-page WebGPU readback path it runs on — so train and live are one distribution by construction. Runs live at /research/vla.
- Input: 3 stacked 48x48 RGB frames (newest first, 9ch) — no joint angles, no coordinates.
- Output: 3 joint deltas. 186,099 params, plain-JS forward, ~9 ms read+infer.
- Data: readback frames = DART (expert perturbed so the arm visits off-distribution states; the expert's clean action is the label) + one DAgger round with task variety forced during capture.
Result
Driving from pixels alone at ~30 Hz it completes ~23 reaches/min — the state-based expert checkpoint it was cloned from does ~21. It matches its teacher from the rendered picture alone.
The correction this artifact exists to record
For most of its development this policy scored ~1 reach/min and looked like a near-failure. The cause was not the policy — it was the control rate. The demo re-perceived every 90 ms (an arbitrary constant), so the policy decided at ~11 Hz while the expert re-decided every frame at ~60; its action went stale and the arm overshot. Dropping the interval to 30 ms (still ~9 ms of real work) took the rate from ~1 to ~23/min.
That one number had silently poisoned every comparison run through it. Measured at 11 Hz, different training runs scored 0–1/min and we built causal stories on the noise ("DAgger collapse", "v5 is the peak"). Re-measured at 30 Hz they are indistinguishable — v5, v6 (which we had called "collapsed"), and v8 all land ~22–23/min. What genuinely mattered: the readback train==live capture + DART lifted a screenshot-trained baseline from ~10/min to parity. The extra DAgger rounds we had credited moved nothing.
| control rate | screenshot-trained (v1) | readback + DART (v8) |
|---|---|---|
| ~11 Hz (the buggy default) | ~0.3/min | ~1/min |
| ~30 Hz (correct) | ~9.7/min | ~23/min |
Method lesson: held-out MSE and expert-agreement were blind to all of this — they barely moved while the true rate went 0 → 23. Only end-to-end rollouts, at a sane control rate, rank these models.
Released CC-BY-4.0 by the Institute for Physical AI @ BMI.