G1 Dual vs. Unified Critic — Loco-Manipulation Policies

PPO policies for Unitree G1 whole-body loco-manipulation, released as the reference checkpoints for the paper Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation (RL4IL @ ICRA 2026), available at https://arxiv.org/abs/2606.11891.

Training and evaluation code: https://github.com/mturan33/isaac-g1-ulc

Checkpoints

The S6u checkpoint was produced by train_ulc_stage_6_unified.py as it stood at commit c545e8a, restored in the code repository as train/23dof/train_ulc_stage_6_unified_asrun_c545e8a.py. The file of the original name at the tip of that repository builds a dual critic and does not reproduce this checkpoint.

* Zero-based final level index over the length of that run's own schedule, as in Table II of the paper (v2 and v3); S6s and S7 reached their terminal level. The three schedules are not on a common scale.

Model details

Action space — 17 joints (benchmark_s6_vs_s7.py:120-135, 777-786). Order is actions[:, :12] legs then actions[:, 12:17] arm. Targets are residuals on the default pose: leg_target = default_leg + a_leg * 0.4, arm_target = default_arm + a_arm * 0.5. In this benchmark the seven right-hand finger joints are held open at their lower limit and are not controlled; S6u's arm actor nevertheless emits seven finger outputs and trained driving them (see the caveats and the re-evaluation below).

• Legs (12): left/right_hip_pitch, left/right_hip_roll, left/right_hip_yaw, left/right_knee, left/right_ankle_pitch, left/right_ankle_roll — interleaved left, right.
• Arm (5): right_shoulder_pitch, right_shoulder_roll, right_shoulder_yaw, right_elbow_pitch, right_elbow_roll.

These 17 controlled joints are a subset of the G1's 23 active body DoF; the seven Dex3 finger joints lie outside that count of 23.

Locomotion observation — 57-dim (benchmark_s6_vs_s7.py:613-643), clamped to ±10:

Arm observation — 52-dim (benchmark_s6_vs_s7.py:717-740), clamped to ±10:

Index 46:49 differs by policy (benchmark_s6_vs_s7.py:706-710): unified_critic_s6u.pt expects zeros there, dual_critic_s6s.pt and dual_critic_antigaming_s7.pt expect the target palm-direction unit vector.

dual_critic_antigaming_s7.pt uses a 55-dim arm observation — the 52 above plus steps_since_spawn / max_steps, end-effector displacement since spawn, and initial_distance / 0.5 (benchmark_s6_vs_s7.py:759-775).

Networks. Locomotion actor/critic [512, 256, 128] with LayerNorm + ELU; arm actor/critic [256, 256, 128] with ELU. The dual-critic checkpoints hold four modules (loco_actor, loco_critic, arm_actor, arm_critic); the unified checkpoint holds three (loco_actor, arm_actor, and a single critic taking the concatenated 109-dim observation). Note that unified_critic_s6u.pt has a 12-dim arm actor head, of which only the first five outputs drive joints — a confound acknowledged in the paper.

Requirements

Isaac Lab 2.3.1 · Isaac Sim 5.1.0 · PyTorch 2.7.0+cu128 · Python 3.11

Usage

import torch

ckpt = torch.load("dual_critic_s6s.pt", map_location="cpu", weights_only=False)

print(ckpt.keys())
# dict_keys(['model', 'iteration', 'best_reward', 'curriculum_level', 'total_reaches'])
print(ckpt["iteration"], ckpt["curriculum_level"])   # 19730 12

sd = ckpt["model"]                       # 46 tensors (dual) / 38 (unified)
print(sd["loco_critic.net.0.weight"].shape)   # torch.Size([512, 57])
print(sd["arm_critic.net.0.weight"].shape)    # torch.Size([256, 52])  (55 for S7)
print(sd["arm_actor.log_std"].shape)          # torch.Size([5])

The unified checkpoint uses different module names — loco_actor.actor.*, arm_actor.actor.* and critic.critic.0.weight of shape [512, 109]. dual_critic_antigaming_s7.pt additionally stores validated_reaches, timed_out_targets and total_attempts.

Weights are plain state_dicts; instantiate the actor/critic classes from test/benchmark_s6_vs_s7.py in the code repository and call load_state_dict.

Results

Standardized benchmark — 3,000 steps, 1 environment, deterministic actions, standing mode.

The original benchmark pins the fingers open for every policy, whereas S6u had trained driving its own. ‡ Same S6u checkpoint, benchmark, seed and step count; the only change is that its seven finger outputs drive the fingers. With that change the standing-mode gap to S6s falls from 3.5x to 1.3x in speed and from 2x to 1.1x in throughput (walking: 2.3x to 1.2x, and 1.8x to 1.03x). This is a single re-evaluation of a single checkpoint, and we do not generalize from it; S6u still runs on the dual run's locomotion branch, not its own, and that second mismatch is unmeasured.

The anti-gaming variant (S7) is faster to target than S6s (5.8 vs. 6.5 steps) but did not raise validated reach rate or throughput. Walking-mode numbers: Table II of the paper.

Known pitfall

Two unified-critic run directories exist from the same day. unified_critic_s6u.pt here comes from ulc_g1_stage6_complete_2026-01-31_20-49-39 (iteration 20,000, curriculum level 10), the run behind the paper's 53.8%.

A second directory, ulc_g1_stage6_complete_2026-01-31_18-54-44, is a false start that was aborted after 22 iterations at curriculum level 0. Its model_best.pt is an untrained snapshot and benchmarks at 0% reach rate with 100% timeouts. It is deliberately not published here.

Experimental caveats

Simulation only. The headline comparison rests on a single pair of training runs that differ in more than the critic.

Held constant between S6u and S6s: robot and simulator; the 17 driven joints; observation spaces (57-dim locomotion, 52-dim arm); network sizes; PPO settings (lr 3e-4 cosine, γ = 0.99, λ = 0.95, clip 0.2, 24-step rollouts); 20,000 iterations; and the evaluation harness — one benchmark script, one locomotion branch given to all three policies (the dual run's, which S6u did not train with; see below), seed 42, and, in the original, pinned-finger evaluation, fingers pinned open for every policy.

Two of those items cut against S6u. The shared locomotion branch is the dual run's: S6s and S7 carry bit-identical locomotion weights, S6u's differ (largest absolute difference 3.38), and the harness replaces S6u's branch at load time unless --s6u_own_loco is passed. Pinning the fingers open matches how S6s and S7 trained but not how S6u trained. The PPO settings are shared; the PPO update rule is not (see below).

Parallel environments. S6s ran at 2048 and S7 at 4096, both recorded. S6u's launch flags were never saved; it is inferred at 2048. Wall-clock cannot decide it (the known-2048 and known-4096 runs take 3.39 and 3.22 s/iteration — iteration cost does not scale with environment count here), but the reach counter can: a run logs at most num_envs × 24 reaches per iteration, so the counter's peak bounds num_envs from below. Across all 20,000 iterations of the S6u run the largest increment is 49,046 — a hard floor of num_envs ≥ 2044, exceeding the 1024 ceiling on 69 iterations, and reaching 99.78% of the 2048 ceiling (49,152) without ever crossing it. 4096 would require the counter to sit at 49.9% of ceiling for 20,000 iterations and never cross half. Consistent only with 2048 — but an inference, not a record. Version 1 of the paper's "4096 for all experiments" held only for S7; version 2 reports the per-run counts.

Curriculum — the two runs did not finish on the same task. This is the largest caveat. The task each policy was training on when its checkpoint was saved:

The 40-level scheme is not a finer-grained version of the 13-level one: it ramps reaching (0–9) → orientation and gripper together (10–19) → commanded torso height (20–29) → payload (30–39), axes S6s never trains, it resets base motion to standing at each phase boundary and re-ramps (its level 9 already commands 0.60 m/s, level 10 drops back to 0), and it only ever trains a fixed palm-down orientation. S6u did enter its gripper phase: Curriculum/level first reaches 10 at iteration 1,162 and stays there, 18,838 of 20,000 iterations. Mapping level 10/40 onto the other ladder: on base motion it sits at S6s levels 0–4, on orientation near S6s levels 7–8 — nowhere near level 12.

S6u was therefore evaluated as a policy that had progressed less far, not one that had reached comparable task difficulty. Two things temper this without rescuing the causal claim. The benchmark's 0.06 m position tolerance is looser than the 0.05 m S6u faced at level 10, and its targets are sampled 0.18–0.40 m from the shoulder, a band S6u had trained on, though the band beyond 0.35 m was sampled only at levels 5–9 (up to 0.38 m at level 5), iterations 327–1,161 of 20,000; level 10 samples 0.18–0.35 m. And both runs used 2048 parallel environments and 20,000 training iterations; for S6u these values are reconstructed from run artifacts (the saved checkpoints and the per-iteration reach counter), because its launch configuration was not recorded. (Earlier versions of this card quoted 0.18–0.28 m for level 10: that is the level's arm_radius, which the absolute-mode target sampler never reads.) Version 1 of the paper reported "Level 10/12"; version 2 corrects it to 10/40.

Also not held constant:

• Arm action dimensionality and finger control. arm_actor.log_std is (12,) for S6u (5 arm
  • 7 finger) and (5,) for S6s/S7. Finger control activates at curriculum level 10, so S6u's seven finger outputs drove the robot for 18,838 of its 20,000 iterations, and they are discarded in the original, pinned-finger evaluation, where the benchmark pins the fingers open for every policy (arm_out[:, :5]). S6u is therefore evaluated in a configuration it did not train in, and the confound is in the task performed as well as in exploration and entropy. Driving its own fingers, with nothing else changed, shrinks the standing-mode gap from 3.5x to 1.3x in speed and from 2x to 1.1x in throughput (one re-evaluation; see Results).
• Reward table. The two scripts do not share a reward table. Forward velocity tracking is vx = 2.5 (unified) versus 5.0 (simplified), and the term lists differ: the simplified configuration adds leg posture, standing still and foot stability; the unified one adds gripper (5.0, non-zero for 18,837 iterations), height tracking (3.0) and load stability (2.0). On the arm side only the distance weight (4.0) is common. Active for the whole of both runs.
• Update rule. The unified script normalizes one summed advantage, applies a single likelihood ratio to both actors and clips gradients over the whole network with one optimizer. The simplified script forms a separate advantage per reward stream, normalizes each, and updates each actor with its own ratio, clipping and optimizer. Critic architecture and credit assignment move together between these runs, and nothing here separates them.
• Locomotion branch at evaluation. The harness gives all three policies the locomotion branch the dual runs share; S6u's own locomotion weights differ from it, so its arm is evaluated with a locomotion branch it did not train with. What that costs has not been measured, and the finger re-evaluation leaves it in place.

Seeding. Neither run seeded anything — at the time of these runs there was no torch.manual_seed, np.random.seed, cfg.seed or --seed in the training scripts, and Isaac Lab leaves the seed unset by default. (A --seed argument was added to train/23dof/ in July 2026, after these runs.) The scripts do draw from the RNG, so every number here comes from one non-deterministic run with no variance estimate. The benchmark seeds itself (default 42, re-applied per policy), so policies are compared on matched target sequences.

What survives. The evaluation is internally consistent — one harness, one shared locomotion branch, matched target sequences — so the measured gap is a real property of these two checkpoints under this harness. It depends on the harness: with S6u's finger outputs restored and nothing else changed, the standing-mode gap falls from 3.5x to 1.3x in speed and from 2x to 1.1x in throughput. For the difference that remains, training progress is at least as parsimonious an explanation as critic architecture, and the unmeasured locomotion-branch mismatch is a third candidate; this design cannot isolate the critic's contribution from the other differences listed above (curriculum ladder, graduation gates, reward tables, action dimensionality, update rule, locomotion branch at evaluation, one unseeded run per arm). Settling it needs a single-variable ablation: one curriculum, one action space, one reward set, one update rule, only the critic swapped, several seeds. That run has not been done; treat these checkpoints as its starting point.

Citation

@article{yardimci2026critic,
  title   = {Critic Architecture Matters: Dual vs. Unified Critics for
             Humanoid Loco-Manipulation},
  author  = {Yard{\i}mc{\i}, Mehmet Turan},
  journal = {arXiv preprint arXiv:2606.11891},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.11891},
  url     = {https://github.com/mturan33/isaac-g1-ulc},
  note    = {ICRA 2026 Workshop on Reinforcement Learning in the Era
             of Imitation Learning (RL4IL)}
}