GR00T N1.7 — Malkist Pick-and-Place (SO-101)
Fine-tune of nvidia/GR00T-N1.7-3B for a
tabletop pick-and-place task on the SO-101 follower arm: picking biscuit ("malkist")
packs and placing them into a drawer box.
- Base model:
nvidia/GR00T-N1.7-3B(~3B-param vision-language-action model) - Embodiment tag:
NEW_EMBODIMENT(SO-101 was not in GR00T pretraining) - Dataset:
monshinawatra/malkist-picknplace - Training step: 5,000
- Task instruction:
Pick all malkists and put them into the drawer box, then reset the arm to rest state.
Task & I/O
| Robot | SO-101 follower, 6-DoF (5 arm joints + gripper) |
| Cameras | front (third-person/overhead) + wrist, 640×480 RGB |
| State | 6-D absolute joint positions: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper |
| Action | 16-step chunk; arm = relative joint deltas, gripper = absolute target |
| Language | natural-language task instruction (see above) |
Camera note: the source dataset stores the third-person view as
observation.images.top; it is mapped to the model'sfrontinput viamodality.json. When deploying, register the overhead camera asfronton the client.
Training
- Method: GR00T N1.7 fine-tuning (Isaac-GR00T). Default tuning — projector + diffusion action head trainable; LLM backbone and vision encoder frozen.
- Data: 50 teleoperation episodes, 27,944 frames @ 30 fps, collected in Isaac Sim with LeIsaac and exported to LeRobot format.
- Modality/data config: SO-100 single-arm config (
examples/SO100/so100_config.py) — state/actionsingle_arm(idx 0–5) +gripper(idx 5–6); videofront+wrist. - Steps: 5,000 (of a planned 10,000; see Limitations).
Usage (remote inference server + thin client)
1. Start the policy server (GPU with 16 GB+ VRAM):
git clone https://github.com/NVIDIA/Isaac-GR00T.git && cd Isaac-GR00T && uv sync
uv run hf download monshinawatra/gr00t-n17-malkist-so101 --local-dir ./ckpt
uv run python gr00t/eval/run_gr00t_server.py \
--model-path ./ckpt \
--embodiment-tag NEW_EMBODIMENT \
--host 0.0.0.0 --port 5555
The checkpoint bundles experiment_cfg/ (modality config + dataset statistics), so no
--modality-config-path / --data-config is needed.
2. Run the client on the machine attached to the SO-101:
cd gr00t/eval/real_robot/SO100
uv run python eval_so100.py \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM0 \
--robot.id=my_follower \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
--policy-host=<SERVER_IP> --policy-port=5555 \
--lang-instruction="Pick all malkists and put them into the drawer box, then reset the arm to rest state."
Open-loop sanity check against the dataset (no robot needed):
uv run python gr00t/eval/open_loop_eval.py \
--dataset-path <lerobot_v2.1_dataset> \
--embodiment-tag NEW_EMBODIMENT --model-path ./ckpt \
--traj-ids 0 --execution-horizon 16 --steps 400
Contents
Standard HF-format GR00T checkpoint: model-0000{1,2,3}-of-00003.safetensors +
model.safetensors.index.json, config.json, experiment_cfg/, embodiment_id.json,
statistics.json, plus training state (optimizer.pt, scheduler.pt, trainer_state.json)
for resuming.
Limitations
- Early checkpoint (5k steps). Trained to half the planned 10k steps; treat as a functional baseline, not a final policy. Resume from the included optimizer state to continue.
- Small dataset (50 episodes) from a single scene/setup — expect limited robustness to new object positions, lighting, backgrounds, or distractors.
- Sim-collected data (Isaac Sim / LeIsaac); a sim-to-real gap may apply on hardware.
- Task-specific to malkist → drawer-box placement with the exact instruction above.
Citation / references
- Base model: NVIDIA Isaac GR00T N1.7 · Isaac-GR00T repo
- Data collection: LeIsaac · LeRobot