GR00T-N1.7 × GEAR-SONIC — prompt-conditioned whole-body motion for Unitree G1 (V2 — cold-start fix)

⚠️ Proof-of-concept — read first

This model memorizes 8 prompts (7 motions + stand), augmented to 54 episodes. Generalization to new prompts / new motions is untested (no held-out split). Demo clips run with deviation/fall terminations relaxed (RELAX=1), so they show prompt → token tracking, not validated balance. It is not a deployable general humanoid VLA. Full caveats in Honest scope & limitations.

What's new in V2 vs V1: V1's one-shot motions (kick / walk / jump) could not self-start from a standing rest pose — the memoryless single-frame policy couldn't tell "about to kick" from "just stood up", so they froze and needed a bootstrap (replay dump tokens to enter the motion). V2 removes the bootstrap: all three one-shots now self-initiate from cold stand (stochastically — see results). The fix is a three-part combination recipe (data + loss + idle), not an architecture change:

1. Physical WBC-rollout augmentation — instead of splicing token frames (non-physical), each [1 s of another motion's momentum] ++ [target onset] token sequence is replayed through the SONIC WBC and re-recorded as (token, ego_rgb, proprio). This covers the deployment "switch-prompt-from-any-state" distribution with physically-valid transitions. Only rollouts that actually launched (not frozen/fell) are kept; all *_from_kick sources are dropped (kick's high foot-lift momentum contaminates the target onset).
2. Onset-loss-weighting — the flow-matching loss upweights the first ~12 token positions, fixing the abrupt high-amplitude kick onset that data alone didn't repair.
3. Stand episodes — 4 "stand" episodes so the policy emits a natural idle instead of drifting into out-of-distribution motion when no action prompt is active.

One prompt-conditioned model. prompt → GR00T → 64-d FSQ token → SONIC.decode → 29-DoF G1 motion. Skills live in the token space; the pretrained SONIC WBC supplies balance/recovery.

🔗 Code, training & validation scaffolding: vitorcen/LeSONIC (scripts/gear_sonic_live_demo.sh, scripts/sonic_build_aug_transitions.py, scripts/sonic_merge_aug_to_lerobot.sh, gear_sonic/data/vla_live_injector.py).

🎥 Closed-loop demos (GR00T-in-the-loop, no C++/DDS)

Each clip is the live closed loop: every control step the policy reads the G1's ego camera + proprioception + the prompt, GR00T predicts a 40-step token chunk over ZMQ, and SONIC's WBC decodes it to joint targets. Zero bootstrap — every motion is driven live from step 0.

⚠️ Clips run with RELAX=1 (deviation/fall terminations at threshold 99 = effectively off), so the full motion plays without the normal 1–2 s deviation reset. They demonstrate prompt → token tracking, not fall-resistance (untested under strict termination).

⭐ Cold-start onset — switch motions at runtime, no restart, no bootstrap

A single persistent live session: the robot stands idle, then the prompt is switched (squat → jump → dance → walk → lunge → kick → macarena …) by writing a control file — the next control step picks it up. This is the V2 headline: walk / jump / kick start from the standing rest pose with no bootstrap injection.

Action sequencing — flow2, all deploy actions chained in one looping session

The prompt is switched over time so the G1 chains every deploy action and loops, with no GUI/server restart; the robot walks to a new spot and performs the next action there. Every segment is pure live (zero replay).

Run it: GR00T_CKPT=<this_checkpoint> bash scripts/gear_sonic_live_demo.sh @flow2.

Results — and how to read them

ℹ️ Measured on the training distribution with no held-out split — these quantify fit, not generalization. Cold-start launch rates are over a small number of trials (n = 2–3) and are stochastic (the memoryless policy has no phase signal, so onset is not deterministic).

Cold-start launch from a standing rest pose — four models, same harness, same RELAX=1, prompt switched at idle (LAUNCH = the motion actually initiates and is not frozen / fallen):

The single-approach models each fix part of the problem (onset-weight → strong kick; augmentation → strong walk); the combination is the first model where all three one-shots launch from cold stand. Honest: still stochastic (~50–100% per motion, not a deterministic 3/3); kick remains the hardest (most abrupt, highest-amplitude single-leg onset).

• Self-sustaining motions (squat, lunge, dance, macarena) run pure live and continue indefinitely, unchanged from V1.
• Open-loop token MSE stays in the same regime as V1 (train-set reconstruction); checkpoint-8000 was selected by the open-loop MSE funnel + the closed-loop cold-start check above. There is no validation curve — selection is on training-distribution fit.
• stand idle: the 4 stand episodes are enough for the live sentinel idle, but not enough for a model-driven "stand" prompt (it still drifts) — the live demo injects a captured stand token at idle rather than routing "stand" through the model.

What this is

⚠️ Honest scope & limitations

A proof-of-concept checkpoint, not a general humanoid VLA:

• Training-distribution only — generalization untested. 8 prompts memorized; new-motion / new-prompt / paraphrase generalization not evaluated.
• Memoryless single-frame policy (architectural). GR00T predicts a 40-step chunk from a single obs frame — no history, no phase signal. V2's augmentation + onset-weighting mitigates the cold-start freeze by teaching the onset distribution from many start states, but cannot make it deterministic — that needs history conditioning (n_obs_steps ≥ 2) or a phase input, which is future work.
• FSQ token modelling caveat. The 64-d token is discrete (FSQ, 16 levels/dim); the head regresses it as a continuous flow-matching target and snaps to the grid at deploy. A discrete objective (per-dim CE) is the cleaner formulation (future work).
• Balance is the WBC's, not this model's. "Doesn't fall" in the demos is largely the pretrained SONIC controller; the VLA's contribution is not yet decoupled by ablation.
• Ego camera looks at the ground (pelvis-mounted, forward); the visual signal for pose is weak — the policy leans on proprioception + prompt.
• Not runnable standalone. Requires the GEAR-SONIC WBC checkpoint + the unitree_g1_sonic embodiment config + the live injector + a running GR00T ZMQ server (GR00T and SONIC can't share a process — transformers version conflict). See the repo for env, ZMQ wire schema, and eval commands.

How to run (closed loop)

# 1. start the GR00T inference server (Isaac-GR00T venv) pointed at this checkpoint
python -m gr00t.eval.run_gr00t_server \
    --model_path <this_checkpoint> --embodiment_tag unitree_g1_sonic --port 5555

# 2. drive SONIC's WBC live in the Isaac viewer (isaaclab env) — zero bootstrap
GR00T_CKPT=<this_checkpoint> bash scripts/gear_sonic_live_demo.sh @flow2   # chained loop
bash scripts/sonic_say.sh kick   # switch the running session to any motion at runtime

Architecture

prompt ─┐
ego cam ─┼─► GR00T N1.7 ──(ZMQ)──► 64-d motion_token ──► SONIC WBC decode ──► 29-DoF G1 (balanced)
proprio ─┘     (one model)                                (frozen controller)

Citation / links

• GEAR-SONIC WBC: https://github.com/NVlabs/GR00T-WholeBodyControl
• NVIDIA Isaac GR00T: https://github.com/NVIDIA/Isaac-GR00T
• Training & augmentation code: vitorcen/LeSONIC