GR00T N1.7 — Unitree G1 bimanual table assembly (RELATIVE end-effector, 60k)
Fine-tune of nvidia/GR00T-N1.7-3B for a bimanual children's-table assembly task on a Unitree G1
with Dex1-1 grippers. Inference-only bundle (optimizer states excluded).
- Dataset:
BitRobot/G1_WBT_Dex1_Building-Children-Table— 533 episodes, LeRobot v3.0, 30 fps, converted here to a GR00T-flavoured v2.1 set at 10 fps (2,092,332 frames), split 479 train / 54 val - Checkpoint: step 60,000 (final; the schedule is cosine so this is the intended end point)
- Training: 4×A100-80GB, 60.3 h, effective batch 256 (per-device 64 × 4 GPUs × accum 1), lr 1e-4 cosine, warmup 5 %, weight decay 1e-5, bf16 + tf32, DeepSpeed ZeRO-2, 70.8 samples/s. VLM backbone (Cosmos-Reason2-2B) frozen — only projector, diffusion head, linear and vlln layers are trained. Train loss 1.123 → 0.0137.
Action / observation space
| modality | keys | dims |
|---|---|---|
| state | ee_left, ee_right, hand | 6 + 6 + 2 = 14 |
| action | same, 16-step chunk (1.6 s at 10 fps) | 14 |
| video | ego_view, right_wrist_view, left_wrist_view | 3 × 480×640 RGB |
| language | annotation.human.task_description | 1 |
End-effector values are body-frame (x forward, y left, z up; metres for position, radians for roll/pitch/yaw), gripper aperture in Dex1 units clamped to [0, 4.5]. Arms use a RELATIVE representation — the policy predicts deltas internally, and the server reconstructs absolute targets against the current state before returning them, so clients always receive absolute poses. The gripper is ABSOLUTE.
⚠️ The origin link of the body frame (pelvis vs torso) is not documented upstream and was not independently confirmed; axis directions were verified empirically from the data. Confirm the frame convention against your controller before deployment.
⚠️ This policy is subtask-conditioned, not a single-instruction policy
The dataset carries a per-frame subtask label, and the model learned to condition on it. The label set is:
insert table leg to table base · move to table · flip table · rotate leg to tighten ·
pick table leg · rotate table base · building children table · move table base
building children table is not a global instruction covering the whole episode — measured over
80 episodes it appears only in the opening 1–6 s of each one, and the rest of the episode cycles
through [move to table →] pick → insert → rotate-leg-to-tighten → rotate-table-base, once per leg.
98.8 % of frames therefore carry a specific subtask label.
Consequence for deployment: feeding a single overall instruction puts the policy far outside its training conditioning. You need an instruction scheduler (a state machine or a higher-level planner) that switches the string as the task progresses, using the exact label strings above.
Execution
Returned chunks are 16 steps at 10 fps. Executing roughly half a chunk (8 steps, 0.8 s) before
re-observing is a reasonable default; a shorter execution horizon in contact-rich phases and a longer
one in free space is the natural refinement. With an end-to-end latency of ~160 ms the constraint
execution_steps × 100 ms > latency is satisfied from 2 steps upward, so the horizon can be tuned
freely.
python gr00t/eval/run_gr00t_server.py \
--model-path <this repo> --embodiment-tag new_embodiment --port 5555
Validated against Isaac-GR00T at commit 65cc4a192, transformers 4.57.3, torch 2.7.1+cu128.
processor_config.json and statistics.json in this repo carry the modality config and
normalization statistics, so no external config import is needed to serve the model. Reading the
gated nvidia/Cosmos-Reason2-2B tokenizer (or a pre-populated offline cache) is required.
Because the vision-language backbone is frozen, deployed camera placement, field of view and exposure should match the training distribution; large visual shifts degrade performance sharply. This matters particularly for real→sim transfer, where rendered imagery differs from the recorded RGB the model was trained on.
Evaluation
Open-loop evaluation on the held-out 54 episodes is running and the numbers will be added here.
For reference, an earlier run of the same recipe on different hardware (2×B200, per-device 16 ×
accum 8, same effective batch 256) reached val54 MSE 0.0409 / MAE 0.0829 at 60k, with per-subtask
MSE ranging from 0.0189 (rotate leg to tighten, the most frequent subtask) to 0.0757
(building children table, the rarest). The weakest subtasks are consistently the least frequent
ones, so additional data on those segments is the main expected lever.
Open-loop error is a proxy. Closed-loop evaluation on the robot is the real test, and open-loop metrics should not be used on their own to rank candidates.