hyundai-uiwang-left-vinn
Non-parametric VINN policy for the Hyundai-Uiwang left arm (HDR35_20 6-DoF + DG5F 20-DoF hand,
26-d action, 30 fps). Repo root is a drop-in AutoModel encoder; banks/ holds the kNN serving
banks; code/ holds the offline eval scripts that produced the tables below.
- Encoder (root, current =
infonce_mask12_left_e6): DINOv3-ViT-S/16 fine-tuned with InfoNCE (symmetric NT-Xent per camera, τ=0.2, bs=128, projector→256d, last 2 blocks unfrozen) on 105 train episodes (stride-3 frames), epoch-6 of a 12-epoch run. Positives = two independent augmentations of the same frame. Same-episode frames within ±45 frames are masked out of the negatives (near-duplicate false negatives at 30 fps). Robustness augmentations on front views, both p=0.5: background-swap (hand+hook-mask background replaced by another episode's frame) and border-band clutter (1–3 random crops pasted into the outer 25%, center untouched). Trained withleft_infonce.pyinvisuomotor-policy-inference. - Bank
banks/left_infonce.npz: all 131 episodes / 110,568 frames ofNgseo/hyundai-uiwang-left.emb(110568, 768) = L2-normalized concat(front CLS, wrist CLS) from this encoder,act(110568, 26),epi(110568,) — theapp/knn_runner.pyformat. - Idle-filtered bank
banks/left_infonce_mask12_filt3e3.npz: same encoder/format, 72,643 frames (65.7%) — see the filtered-bank section below.
Held-out kNN L1 (105-ep bank → 26-ep query, lower is better)
| encoder | k=1 | k=8 | k=32 |
|---|---|---|---|
| frozen DINOv3 | 0.0913 | 0.0773 | 0.0730 |
| BYOL nomask e6 | 0.0806 | 0.0694 | 0.0651 |
| InfoNCE e2 (previous release) | 0.0759 | 0.0675 | 0.0634 |
| InfoNCE mask12 e6 (this repo) | 0.0770 | 0.0677 | 0.0633 |
Robustness: perturbed-holdout eval (code/left_robust_eval.py)
Real-robot trials showed mask-trained encoders are much better than clean L1 suggests. This eval reproduces that offline: front query images get synthetic perturbations (identical seeded recipes across encoders), bank and wrist stay clean. k=32 L1:
| encoder | clean | clutter | bgshift | clutter+shift |
|---|---|---|---|---|
| nomask e2 | 0.0634 | 0.0665 (+4.9%) | 0.0636 | 0.0667 (+5.2%) |
| mask e6 (bg-swap only) | 0.0633 | 0.0660 (+4.3%) | 0.0631 | 0.0655 (+3.5%) |
| mask12 e6 (this repo) | 0.0635 | 0.0640 (+0.8%) | 0.0634 | 0.0637 (+0.3%) |
The border-clutter augmentation is what buys the ~6× smaller degradation under clutter; the clean-holdout tie hides this entirely. Use this eval alongside clean L1 for encoder A/Bs.
Experimental: mask-latent concat retrieval (code/left_maskcat_eval.py) — null result
Tested appending the hand+hook-mask-pooled DINOv3 patch vector to the retrieval embedding:
normalize(concat(front_CLS, wrist_CLS, α·maskpool)). banks/left_maskcat_stride3.npz holds the
per-frame components for all stride-3 frames of all 131 episodes (cf, cw, mp fp16 + act,
epi, fi; 0% empty masks) so the α sweep is reproducible without re-encoding.
| α | k=8 | k=32 |
|---|---|---|
| 0 (baseline) | 0.0648 | 0.0611 |
| 0.25 / 0.5 | 0.0648 | 0.0611 |
| 1.0 | 0.0650 | 0.0613 |
| 2.0 | 0.0656 | 0.0620 |
Verdict: no gain — the mask-pooled patch feature is redundant with what the (mask-aug-trained) CLS already encodes; heavy weighting only adds noise. Mask value is already absorbed via the training-time augmentations (see robustness table). If retrying, use a genuinely different representation (RF-DETR decoder latent, mask geometry, FK hand pose), not same-encoder pooling. Stride-3-bank numbers are not comparable to the 88k-bank tables above.
Idle-filtered bank (banks/left_infonce_mask12_filt3e3.npz)
Actions are absolute joint targets, so "zero action" = near-zero consecutive-action delta. This
bank drops frames with Linf(Δa) < 3e-3 rad (34.3% — mostly late-episode grasp-hold/settle: 71%
of dropped frames sit in the 60–100% episode span) plus 32 teleop-glitch frames commanding
all-zero arm targets. Built by code/build_left_filtered_bank.py from per-frame motion stats.
| bank | size | holdout k=32 L1 |
|---|---|---|
full (left_infonce.npz) | 110,568 | 0.0633 |
| filtered (this) | 72,643 (−34%) | 0.0632 |
Equal accuracy, ~34% faster kNN matmul, and it removes the visually-aliased hold frames that motivated the loop-break heuristic. Caveats: (1) chunks fast-forward across deleted holds (boundary deltas are tiny by construction; the Ruckig smoother absorbs them); (2) late-episode (60–90%) bank coverage thins ~2× — if the robot rushes grasp/settle on-robot, fall back to the full bank or a run-thinned variant (keep idle-run endpoints; same offline L1).
Variants
variants/nomask_e2/ — the no-mask InfoNCE e2 encoder (previous release; clean L1 0.0634 ties
the champion but no bg/clutter robustness — see the robustness table). Its bank:
banks/left_infonce_nomask_e2.npz. Load with
AutoModel.from_pretrained(repo, subfolder="variants/nomask_e2").
Media
media/mask_overlay_left_ep20.mp4 — episode 20 with the RF-DETR hand+hook mask overlaid (red,
stride-3 frames @10fps): the mask that drives the bg-swap augmentation.
Serve (chunk + temporal-ensembling VINN)
huggingface-cli download Ngseo/hyundai-uiwang-left-vinn --local-dir vinn_left
export VPI_POLICY=knn
export VPI_KNN_DINO=vinn_left # AutoModel dir (repo root)
export VPI_KNN_BANK=vinn_left/banks/left_infonce.npz
# defaults: k=32, VINN-paper softmin-L2 weighting (VPI_KNN_WEIGHT=softmin_l2, VPI_KNN_BW=1.0),
# chunk=8 + temporal ensemble -> single 26-d action per tick
python -m uvicorn app.server:app --host 0.0.0.0 --port 8000