GR00T N1.7 — G1 + Dex3, left-hand pick-and-place (RELATIVE arms)
Fine-tune of nvidia/GR00T-N1.7-3B on real-robot Unitree G1 (Dex3 hands) teleoperation where the
left hand performs the manipulation. Inference-only bundle (optimizer states excluded).
This is the recommended model of the left-hand pair; the all-ABSOLUTE variant is an ablation control and scores 23 % worse.
- Dataset:
carroll511/g1_dex3_nubjuki_left_53ep_260730— 53 episodes / 13,041 frames / 30 fps (7.25 min), single task"pick up the doll and place it in the red region" - Split: 47 train (11,394 frames) / 6 val, every 10th episode held out
- Checkpoint: step 10,000 (best of 2k–10k on the val split)
- Training: 4×A100-80GB, 7.9 h, effective batch 64 (per-device 16 × 4 GPUs × accum 1), ≈56 epochs, lr 1e-4 cosine, bf16, DeepSpeed ZeRO-2. VLM backbone frozen.
Results (val, open-loop, MAE in radians)
| model | arms | overall | left_arm | right_arm | left_hand | right_hand |
|---|---|---|---|---|---|---|
| this | RELATIVE | 0.07161 | 0.06088 | 0.02568 | 0.16733 | 0.03257 |
…-left-abs | ABSOLUTE | 0.09310 | 0.11465 | 0.05607 | 0.16540 | 0.03630 |
RELATIVE arms win by 23 %, and the gap is entirely in the arms (1.9× on the left arm, 2.2× on the right). The left hand — the manipulating hand — is ABSOLUTE in both models and scores essentially the same (0.1673 vs 0.1654), which is the internal control confirming the arm representation is what made the difference.
This replicates the same experiment on the right-hand dataset
(…-nubjuki vs
…-nubjuki-abs), where RELATIVE won
by 26 % with the same signature. Two independent datasets, same conclusion.
The manipulating hand is the hardest group in both datasets (here left_hand at 0.167 rad; in the
right-hand dataset right_hand at 0.096 rad) — fine finger control during the grasp is the
bottleneck, and more data on grasp/re-grasp segments is the main expected lever.
Absolute error here is higher than the right-hand model's (0.0716 vs 0.0455), which is consistent with this dataset being 2.4× smaller (7.25 min vs 17.7 min).
Interface
Observation — 3 RGB cameras at 480×640 (ego_view ← head camera, right_wrist_view,
left_wrist_view), current 28-d joint state in radians, and the task string
"pick up the doll and place it in the red region".
Action — 28-d joint targets in radians, absolute values as returned by the server (arms are modelled as deltas internally and reconstructed against the current state before returning). 16-step chunk intended for 30 Hz execution (33 ms per step); executing about half a chunk before re-observing is a reasonable default.
Joint order (identical for state and action):
| range | group | joints |
|---|---|---|
| 0–6 | left_arm | ShoulderPitch, ShoulderRoll, ShoulderYaw, Elbow, WristRoll, WristPitch, WristYaw |
| 7–13 | right_arm | same seven, same order |
| 14–20 | left_hand | Thumb0, Thumb1, Thumb2, Middle0, Middle1, Index0, Index1 |
| 21–27 | right_hand | Thumb0, Thumb1, Thumb2, Index0, Index1, Middle0, Middle1 |
⚠️ The two hands do not share a finger order — left is Thumb→Middle→Index, right is Thumb→Index→Middle — and their bending signs are mirrored. This comes from the source dataset's own metadata and the model learned it as-is. Mapping both hands with one rule will silently command the wrong fingers.
python gr00t/eval/run_gr00t_server.py --model-path <this repo> --embodiment-tag new_embodiment
Validated against Isaac-GR00T at commit 65cc4a192, transformers 4.57.3, torch 2.7.1+cu128.
processor_config.json and statistics.json carry the modality config and normalization, so no
external config import is needed to serve. Reading the gated nvidia/Cosmos-Reason2-2B tokenizer
(or a pre-populated offline cache) is required.
Because the vision-language backbone is frozen, deployed camera placement, field of view and exposure should match the training distribution. Open-loop error is a proxy; closed-loop evaluation on the robot is the real test.