GR00T N1.7: SO-101 fine-tune on MolmoAct2 community data
LoRA fine-tune of nvidia/GR00T-N1.7-3B (VLM
backbone frozen; adapters on the action head) for the SO-101 follower arm, trained on the
SO-101 subset of allenai/MolmoAct2-SO100_101-Dataset
with MolmoAct2's annotated language instructions. Adapters (55.4M params, 1.73% of 3.2B; via
peft + category-lora)
are merged into this repo's bf16 weights; raw adapters are in adapters/.
Intended use & safety
- Intended use: research and evaluation on SO-101 follower arms (single 6-dof arm, front + wrist cameras) for the tabletop manipulation task families present in the data (pick/place, stacking, sorting, and similar).
- Out of scope: any other embodiment or rig without fine-tuning (VLA policies do not zero-shot transfer across embodiments); unattended operation; operation near people without a hardware e-stop and enforced workspace/torque limits.
- Validation status: offline held-out action loss only (see Losses & evaluation); no sim or real-robot rollouts yet. Users are responsible for safe integration (guardrails, e-stop, workspace limits) before any deployment.
Training
| Data | 39 source repos / 2,242 episodes / 1.8M frames, filtered from the 1,220-repo manifest (rule: robot_type contains so101, identifiable front+wrist cameras, ≥20 episodes, 6-dof state/action). Split: episode-level, 5% held out per repo (min 1), seed 17 → 2,130 train / 112 test episodes |
| Embodiment | NEW_EMBODIMENT with Isaac-GR00T's examples/SO100/so100_config.py; state & action single_arm → [0:5], gripper → [5:6] |
| Image preprocessing | uniform 256×256 letterbox (LongestMaxSize + pad; community repos mix 16:9 and 4:3 cameras), GR00T-default augmentations |
| Schedule | 22,000 steps @ global batch 64 (no accumulation), lr 3e-4 (5% warmup, decay), 1× H100 80GB |
| Checkpoint selection | argmin held-out eval loss over durable checkpoints saved every 250 steps → step 21,000 |
| Headline curve | held-out flow-matching loss: 1.129 → 0.0273 |
LR sweep (250 steps, bs=64, same eval): 5e-5→0.97, 1e-4→0.79, 3e-4→0.59, 6e-4→0.44, 1.2e-3→0.79. 6e-4 won short-horizon but NaN'd the full run at step ~2.5k; 3e-4 trained clean end-to-end.
Losses & evaluation
- Training loss: GR00T N1.7's flow-matching action objective. The DiT action head
predicts the velocity field for a noised 16-step action chunk (conditioned on frozen VLM
features + robot state), MSE against the interpolation target. Trainable: LoRA adapters on
252 action-head Linears + 7
CategorySpecificLinear, plus action-head norms. Frozen: LLM, vision encoder, all base weights. - Eval regime: offline action loss. The same flow-matching objective on the 112 held-out
episodes (episode-level split, 5%/repo, seed 17). 384 cached samples drawn round-robin from
all 39 repos' test splits, processed by the training processor instance (identical
normalization statistics); flow-matching noise/timesteps seeded via
fork_rngso values are comparable across steps and runs. Upstream Isaac-GR00T has no working eval path; ours is implemented in the training repo (src/callbacks.py). - Scope: this is an action-prediction loss on same-task/unseen-trajectory data; it is not a task success rate and does not measure cross-task or cross-scene generalization.
Provenance
| Trained by | Robocurve (jeqcho), 2026-07-06 → 07 |
| Training code | https://github.com/robocurve/gr00t-n1.7-so-101 (public: plan, decision records, incident log in CLAUDE.md) |
| Framework | Isaac-GR00T @ ab88b50c718f6528e1df9dcbaf75865d1b604760; torch 2.7.1+cu128, transformers 4.57.3, peft 0.17.1, category-lora 0a02f398 |
| Compute provider | Modal, 1× NVIDIA H100 80GB; 16 vCPUs / 64 GB RAM (video decode was the dataloading bottleneck) |
| Wall-clock | ~11.5 h across resumed segments; preemptions: 1, max steps lost ≤5 |
| Total compute | ≈4×10¹⁸ FLOPs: torch.utils.flop_counter on a real batch ×3 fwd→fwd+bwd heuristic; order-of-magnitude only (undercounts flash-attn, overcounts the frozen backbone) |
| Cost | ≈$70 (published run) / ≈$130 project-total incl. 7-config sweep and two failed attempts |
| Experiment tracking | Weights & Biases, project gr00t-n17-so101, run main-04 |
| Card authorship | written at publish time by the training session (first-hand), from trainer_state.json, wandb, and the repo's decision records |
Usage
from gr00t.model.policy import Gr00tPolicy
policy = Gr00tPolicy.from_pretrained("robocurve/gr00t-n1.7-so101-molmoact2")
- Observations: cameras
front(third-person) andwrist(gripper-mounted), letterboxed to 256×256; statesingle_arm → [0:5],gripper → [5:6]in LeRobotso101_followerjoint conventions (shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper). - Actions: same 6-dof layout;
single_armuses GR00T's relative-action representation,gripperabsolute (perso100_config.py); horizon 16 steps. Values are normalized with per-dataset q01/q99 statistics shipped inexperiment_cfg/andprocessor/. The policy wrapper applies them; do not feed raw joint values elsewhere. - Serving: Isaac-GR00T's
gr00t/eval/run_gr00t_server.py; SO-101 client:gr00t/eval/real_robot/SO100/eval_so100.py --robot.type=so101_follower. Requires Isaac-GR00T at the pinned commit or later (N1.7).
Data provenance & caveats
- All 39 source repos are public community LeRobot datasets indexed by the Apache-2.0
MolmoAct2 manifest;
each source repo carries its own license; the filtered list with repo ids is
data/repos_filtered.json. - Known data issues: several near-duplicate repos (the same dataset re-uploaded by different
users), which upweights those scenes; task/scene imbalance across repos was tempered with
ds_weights_alpha=0.5. - Eval limits: offline loss only; expect a sim/real gap. No real-robot rollouts yet.
- Instabilities and failed attempts (lr-6e-4 NaN, mixed-aspect-ratio crash, checkpoint-cadence
and GPU-starvation fixes) are documented in the training repo's
CLAUDE.md.
Versioning & contact
mainis stable (single published checkpoint, step 21,000). Pin the revision hash for exact reproduction. Superseding checkpoints will setnew_versionhere.- Issues: HF Discussions on this repo or GitHub issues.