GR00T-N1.7-3B-OMX-PickupDolls-2Cam
2-camera (top-down + wrist) fine-tune of nvidia/GR00T-N1.7-3B for the ROBOTIS Open Manipulator-X "Pick up dolls" task, by Pseudo Lab NVIDIA Robot Foundation Model study group.
This is the two-view successor of pseudolab/GR00T-N1.7-3B-OMX-PickupDolls (single top-down camera). The wrist view was added to address height (z-axis) perception limits observed during real-robot deployment of the single-camera model (gripper closing before reaching the object).
Training
- Base:
nvidia/GR00T-N1.7-3B, embodimentNEW_EMBODIMENT(frozen backbone, DiT action head) - Data: pseudolab/omx_f_PickUpDollWith2Cam
— 90 episodes (4 camera-layout-inconsistent episodes excluded), 60→30 fps resampled,
per-episode camera roles normalized to
camera1=top-down /camera2=wrist - Steps: 18k (global batch 32, H100) — selected via a 5-episode holdout overfitting curve on a 85-episode split (holdout MSE minimum 0.0075 @18k, rebound at 20k), then retrained on all 90 episodes to 18k
- Final open-loop MSE: 0.0034 (general trajectories) / 0.0036 (ex-holdout trajectories)
Observation format (PolicyServer / PolicyClient)
obs = {
"video": {
"camera1": rgb_topdown[None, None], # uint8 (1,1,480,640,3) RGB
"camera2": rgb_wrist[None, None], # uint8 (1,1,480,640,3) RGB
},
"state": {
"single_arm": joints[None, None], # float32 (1,1,5) rad, joint1..joint5
"gripper": grip[None, None], # float32 (1,1,1)
},
"language": {"annotation.human.task_description": [["Pick up dolls"]]},
}
action, _ = client.get_action(obs)
# action["single_arm"][0] -> (16,5) ABSOLUTE joint targets in rad
# (the server's decode_action already converts the model's RELATIVE outputs)
# action["gripper"][0] -> (16,1) absolute (0.0 open .. ~0.69 closed)
Serving
python gr00t/eval/run_gr00t_server.py \
--model-path pseudolab/GR00T-N1.7-3B-OMX-PickupDolls-2Cam \
--embodiment-tag new_embodiment --port 5555
Both cameras must reproduce the collection rig: camera1 fixed top-down
(white 3-slot box at top of frame, dolls below, arm entering from the right),
camera2 mounted on the wrist/gripper.
Files
model-*.safetensors— fine-tuned weights (DiT action head + adapters)experiment_cfg/— training/processor configs and dataset statisticsomx_f_2cam_config.py— modality config used for fine-tuning (video: camera1+camera2)