MiniRT
A small autoregressive vision-language-action policy for the SO-101 arm, and the hierarchy it needs to become selective.
Code, evaluation harness and the full report: https://github.com/charliechin424/MiniRT
Act one. One 96 MB weight drives eight ManiSkill3 manipulation tasks from free-form English at 90.3% mean success over 300 episodes per task.
Act two. The same architecture scores 0% when the task requires choosing — "put the blue can in the bin" with four objects on the table. It reaches, grasps and lifts competently and takes the wrong object; colour grounding is a cross-modal binding it cannot learn at the motor level. Moving that job into a 0.1 M-parameter grounder that shares the policy's own vision backbone recovers 26% end-to-end, with no privileged knowledge of which object is which.
Files
| file | what it is |
|---|---|
v7_mt_generalist_fp16.pt | act one — one weight, eight tasks. 47.9 M params, 96 MB fp16. |
v8_mt_colorsort_fp16.pt | act one plus the ColorSort skill; the warm-start ancestor of the act-two policy. |
colorsort/v15cs_frozen_fp16.pt | act two — the delivered position-conditioned VLA. Frozen vision backbone so the grounder and the policy genuinely share one encoder: 48.1 M total. |
colorsort/v15cs_frozen_int8.pt | the same policy as W8A8 integer compute, 57.7 MB. Costs success — see below. Ships with colorsort/v15cs_frozen_int8_params.json, the per-layer quantisation parameter table. |
colorsort/grounder_r4_mix20.pt | act two — the MiniRT-Grounder heads (0.104 M params on a DINOv2-S backbone). Trained at img_size=448. |
Numbers
Act one, success_at_end over 300 episodes (3 × 100, seed 3000), physx_cpu:
| Task | success |
|---|---|
| ReachCube | 99.3 |
| ReachCan | 99.0 |
| LiftCube | 95.7 |
| StackCube | 93.0 |
| PlaceCube | 91.3 |
| LiftCan | 89.3 |
| PlaceCan | 84.7 |
| StackCan | 70.3 |
| mean | 90.3 |
Unseen instruction phrasings: seven of eight tasks show no degradation.
Act two, 100 episodes, seed 3000:
| privileged ceiling | grounded closed loop | n=2 | n=3 | |
|---|---|---|---|---|
| frozen 48.1 M (delivered) | 29% | 26% | 41% | 10% |
Perception costs three points; insisting one encoder serve both the grounder and the policy costs about half the success rate. That is the trade-off, stated rather than hidden.
Running these
The evaluation is not reproducible from the weights alone. It needs the ColorSort environment,
which is a modification to an upstream repository and ships in the code repo as squint_patch/, and
it needs several flags whose defaults are wrong for these checkpoints — --min-area 6 in
particular silently costs eleven points if left at its default. The README has the exact commands:
https://github.com/charliechin424/MiniRT
Notes
- Every number is measured on
physx_cpu.physx_gpudiverges physically, per task and in either direction; do not compare across backends. - The act-two checkpoint has catastrophically forgotten the eight base tasks. Its ancestor
v8_mt_colorsorthas not. This is a design consequence — six generations of fine-tuning with zero base-task replay — not a defect, and it is recorded here so that comparing the two checkpoints does not lead to the wrong conclusion. - The int8 build costs success and is labelled so. fp16 26/100, int8 22/100; at the object level 58% against 47% over 249 objects. Both acceptance gates still pass, so it is usable — choose it for the 40% size reduction, not because it is free. The four-point headline gap sits on the noise band of a 100-episode run and read alone would suggest no effect; the paired per-seed comparison (identical scenes and wording) shows int8 worse on 37 seeds and better on 17, p ~ 0.009. On the single-task predecessor quantisation was genuinely free, which is why this is worth stating.
Credit
Built on squint (SO-101 ManiSkill3 environments), ManiSkill3, LeRobot and DINOv2.