GR00T N1.7 — Unitree G1 + Dex3, "pick up the doll and place it in the red region"
Fine-tune of nvidia/GR00T-N1.7-3B on real-robot Unitree G1 (Dex3 hands) teleoperation data.
Inference-only bundle (optimizer states excluded).
- Dataset:
carroll511/g1_dex3_nubjuki— 100 episodes / 31,803 frames / 30 fps (17.7 min total), single task - Checkpoint: step 10,000 (best of 2k/4k/6k/8k/10k on a held-out 10-episode val split)
- Training: 4×A100-80GB, 7.5 h, effective batch 64, ≈22 epochs, cosine lr 1e-4, bf16, DeepSpeed ZeRO-2. VLM backbone (Cosmos-Reason2-2B) frozen; only projector + diffusion head + linear + vlln trained.
Results — open-loop on val (10 episodes, action horizon 16)
| checkpoint | MAE (rad) | MSE | left_arm | right_arm | left_hand | right_hand |
|---|---|---|---|---|---|---|
| 10000 | 0.04553 | 0.011323 | 0.01700 | 0.04844 | 0.02064 | 0.09605 |
| 8000 | 0.04606 | 0.011324 | 0.01774 | 0.04996 | 0.02117 | 0.09535 |
| 6000 | 0.04777 | 0.011343 | 0.01742 | 0.05334 | 0.02146 | 0.09886 |
| 4000 | 0.05172 | 0.012980 | 0.01943 | 0.05798 | 0.02335 | 0.10612 |
| 2000 | 0.06524 | 0.020763 | 0.02489 | 0.07174 | 0.02459 | 0.13975 |
Overall MAE 0.0455 rad = 2.61°. No overfitting was observed through 10k, but improvement is exhausted (MSE at 6k/8k/10k is 0.01134/0.01132/0.01132 — statistically indistinguishable); more training steps are unlikely to help, more data will.
Hardest group is the right hand (the grasping hand) both absolutely (5.50°) and relative to how much it moves (MAE/std 0.22 vs 0.10–0.17 for the others). The left arm/hand move little in this task, so their low error reflects task difficulty, not model quality.
Open-loop error is a proxy only — real verdict requires closed-loop evaluation on the robot.
Interface contract (fixed at training time — cannot be changed without retraining)
Observation
- 3 RGB cameras, 480×640:
ego_view← head camera (the dataset'scam_right_high),right_wrist_view←cam_right_wrist,left_wrist_view←cam_left_wrist. (The dataset'scam_left_highis a near-duplicate stereo twin ofcam_right_highand is unused.) - Current joint state, 28-d float32, radians (order below)
- Language: the exact training string
"pick up the doll and place it in the red region."
Action — 28-d joint targets, radians, absolute values returned by the server (arms are modelled as deltas internally but reconstructed against the current state before returning). Chunk of 16 steps intended to run at 30 Hz (33 ms per step). Re-planning after executing about half the chunk (8 steps) is recommended.
Joint order (identical for state and action)
| idx | joint | idx | joint |
|---|---|---|---|
| 0–6 | left_arm: ShoulderPitch, ShoulderRoll, ShoulderYaw, Elbow, WristRoll, WristPitch, WristYaw | 14–20 | left_hand: Thumb0, Thumb1, Thumb2, Middle0, Middle1, Index0, Index1 |
| 7–13 | right_arm: same 7 in the same order | 21–27 | right_hand: Thumb0, Thumb1, Thumb2, Index0, Index1, Middle0, Middle1 |
⚠️ The two hands do not use the same finger order. Left is Thumb→Middle→Index, right is Thumb→Index→Middle. This comes from the source dataset's own metadata and the model learned it as-is. Bending sign is also mirrored: the left hand's middle/index flex toward negative values, the right hand's toward positive. Mapping both hands with one rule will silently command the wrong fingers.
Usage
python gr00t/eval/run_gr00t_server.py \
--model-path <this repo> \
--embodiment-tag new_embodiment \
--port 5555
Requires Isaac-GR00T (validated at commit 65cc4a192), transformers 4.57.3, torch 2.7.1+cu128,
and read access to the gated nvidia/Cosmos-Reason2-2B (or a pre-populated offline HF cache).
processor_config.json / statistics.json in this repo carry the normalization and modality
config, so no external config import is needed to serve the model.
Because the vision-language backbone is frozen, the deployed camera placement, field of view and exposure should match the training data; large visual shifts degrade performance sharply.