HIL-SERL offline prep bundle — Put the right banana in the pot (UR7e)

This repository is the offline-prepared base for HIL-SERL (Human-in-the-Loop Sample-Efficient Robotic reinforcement Learning) on the "put the right banana in the pot" task with a Universal Robots UR7e. It contains everything up to online RL — the vision reward classifier, the SAC/RLPD training config, the UR7e kinematics (URDF + FK), the offline demo buffer builder, and the runbook. The online phase itself is robot-only and is not included here (it requires the physical arm plus ur_rtde + placo + a gRPC actor/learner).

Built from Bigenlight/banana_in_pot_lerobot_v3 (51 teleop demos / 21,524 frames) using LeRobot's HIL-SERL stack.

Data & hardware setup

What's in this bundle

banana_in_pot_hilserl/
├── reward_classifier/
│   └── checkpoint/              # trained success classifier (config.json + model.safetensors)
├── config/
│   └── train_hilserl_ur7e.json  # shared SAC/RLPD learner+actor config (paths relative)
├── build_offline_buffer.py      # rebuilds the 54 MB RL demo buffer from the dataset
├── joint_to_ee.py               # UR7e forward kinematics (Joint → EE), placo-free, validated
├── ur7e.urdf                    # pre-generated UR7e URDF (IK target frame `tool0`)
├── HILSERL_RUNBOOK.md           # exact online start sequence (robot-only steps marked [ROBOT])
└── assets/                      # figures embedded below

Offline demo buffer is NOT shipped (it is ~54 MB of video/state transitions). Rebuild it from the dataset with build_offline_buffer.py — see Rebuilding the demo buffer.

1. Reward / success classifier

A vision-based binary success detector trained offline, in the exact LeRobot reward_classifier format so it drops straight into HIL-SERL.

• Encoder: frozen lerobot/resnet10 (CNN), per-camera SpatialLearnedEmbeddings pool + Linear→LayerNorm→Tanh. Following the official LeRobot behavior, the encoder runs under no_grad so only the classifier head trains (~2.36 M trainable params).
• Cameras: observation.images.cam1 + observation.images.cam2, 128×128, MEAN_STD-normalized.
• Labeling (all 51 demos are successes, so negatives are synthesized): POSITIVE = last 15% of frames and gripper re-opened (release into pot); NEGATIVE = first 55% of frames; the 55–85% transport margin is excluded. Split by episode (val = eps [4,14,24,34,44]).
• Deployed with success_threshold = 0.7 for release-boundary margin.

2. Action spec — EE-delta

The demos are absolute joints, but the HIL-SERL policy acts in end-effector delta space.

• Action = base-frame TCP delta ÷ step_size (0.05 m), computed by running FK on the dataset joints (identical to what online deploy uses as its per-step reference). The gripper is a discrete class {0=close, 1=stay, 2=open}.
• The offline buffer stores a 4-dim action (continuous xyz + discrete gripper); the SAC critic slices actions[:, :-1] for the continuous part and a separate discrete critic handles the gripper. Hence output_features["action"].shape = [3] with num_discrete_actions = 3.
• Action stats (21,524 frames): p99 ≈ 0.2 in tanh space, |Δ|>1 = 0.0% — comfortably inside the [-1, 1] range.

Joint → EE forward kinematics (validated)

joint_to_ee.py implements UR7e FK directly from the URDF joint origins (placo-free) and was validated against the recorded TCP pose:

Sub-mm / sub-0.2° error while static confirms the kinematic chain is accurate; the larger moving-sample error is timing jitter between the two async logging streams (joint vs. TCP), not an FK error. Online deploy IK uses placo (Pinocchio), frame tool0.

3. Training config (SAC + RLPD)

config/train_hilserl_ur7e.json is the shared TrainRLServerPipelineConfig for the learner and actor, fully consistent with the offline buffer:

The learner setup was validated end-to-end offline: python -m lerobot.rl.learner builds the SAC policy (2.76 M trainable / 7.67 M total params), loads the offline demo buffer via ReplayBuffer.from_lerobot_dataset, starts its gRPC server, and idles at the online-buffer gate (len(online_buffer) < online_step_before_learning=100) waiting for the actor — which is exactly the "ready for online RL" state.

4. The online phase (robot-only, NOT in this repo)

See HILSERL_RUNBOOK.md for the exact sequence. In short, on the machine wired to the arm:

1. Install the hardware/transport deps that the offline env intentionally omits: grpcio (py3.12 wheel), placo (IK), ur_rtde (UR I/O), lerobot[hilserl].
2. Fill env.robot / env.teleop in the config (UR7e IP + motors + cameras; gamepad teleop).
3. Tune the workspace placeholders against the real arm: end_effector_bounds, fixed_reset_joint_positions, episode length. Keep end_effector_step_sizes = 0.05 — the offline action = TCP-delta ÷ 0.05, so changing it desyncs the demo actions.
4. Start the learner (terminal 1) and actor (terminal 2) with the same config. The learner idles at the gate until the actor supplies ≥100 online transitions, then SAC updates begin with the 50/50 RLPD mix.
5. Human interventions: gamepad trigger / space to take over and give corrective demos; taper the intervention rate as the policy improves.

RAM note: the full offline buffer (from_lerobot_dataset) materializes every transition eagerly and peaks at ~25 GB. Use ≥48 GB RAM, or subset episodes via --dataset.episodes='[...]', or lower offline_buffer_capacity.

Rebuilding the offline demo buffer

The 54 MB RL demo buffer is not shipped. Rebuild it from the LeRobot dataset with the included script (it converts absolute-joint demos to the EE-delta + reward + done schema):

python build_offline_buffer.py   # writes ./banana_rl_lerobot (repo_id theo/banana_in_pot_rl)

Then point dataset.root in config/train_hilserl_ur7e.json at the rebuilt directory (the config ships with a relative ./banana_rl_lerobot).

Related repositories

• Dataset: Bigenlight/banana_in_pot_lerobot_v3
• ACT imitation-learning policy for the same task: Bigenlight/act_banana_in_pot

Caveats & limitations

• No true failure episodes. The classifier's negatives are early-task frames (approach/grasp), not genuine failed attempts. It reliably separates "task complete" from "task in progress" but has never seen a real failure of a completed-looking state — expect over-confidence on OOD near-miss end states. Record a handful of real failure/near-miss episodes early in online HIL-SERL to harden it.
• The excluded 55–85% transport margin means the decision boundary around the release moment is uncalibrated; success_threshold=0.7 adds margin.
• EE safety bounds and reset pose in the config are placeholders — they MUST be tuned against the real arm before letting the policy move.
• Only the classifier head trains (frozen-encoder LeRobot quirk) — fine for this easy visual task, the first thing to revisit if a harder task underperforms.
• All validation here is offline (config parse → policy/critic build → buffer load → gate). No online RL results are included; that is the robot-only next step.