SAC + HER agent playing PandaPickAndPlace-v3
This is a SAC (Soft Actor-Critic) agent trained with Hindsight Experience Replay (HER)
on the PandaPickAndPlace-v3 environment
from panda-gym, using
stable-baselines3.
Why HER?
PandaPickAndPlace-v3 is a sparse-reward, goal-conditioned task: the agent gets
-1 reward every step until the object reaches the target position, 0 otherwise.
Plain off-policy RL (SAC/TD3/DDPG) essentially never learns here, because a random
policy almost never reaches the goal, so almost every trajectory carries the same
uninformative reward signal.
HER fixes this by relabeling failed trajectories: for a rollout that failed to reach
its intended goal g but happened to move the object to some other position g',
HER stores an extra transition where g' is treated as if it had been the goal all
along, with the reward recomputed accordingly. This turns every episode into a
source of positive learning signal, "success at a different goal", even when the
original goal wasn't reached.
Results
Evaluated over multiple episodes with deterministic actions:
Mean reward = -20.80 +/- 19.18
(episode length capped at 50 steps, -50 = never reached goal, values closer to
0 = faster successful pick-and-place)
Demo
Training hyperparameters
| Hyperparameter | Value |
|---|---|
| Algorithm | SAC |
| Replay buffer | HerReplayBuffer |
n_sampled_goal | 4 |
goal_selection_strategy | future |
learning_starts | 1000 |
learning_rate | 1e-3 |
buffer_size | 1,000,000 |
batch_size | 1024 |
gamma | 0.95 |
tau | 0.05 |
net_arch | [512, 512, 512] |
n_critics | 2 |
n_envs | 8 |
| Total timesteps | 1,000,000 |
| Observation normalization | VecNormalize(norm_obs=True, norm_reward=False) |
learning_starts must exceed the environment's max episode length (50 steps) so
that HER always has at least one complete episode to relabel before the first
gradient update — otherwise SB3 raises RuntimeError: Unable to sample before the end of the first episode.
Full training script: train.py.
Usage (with Stable-Baselines3)
pip install stable-baselines3 sb3-contrib huggingface_sb3 panda-gym
import gymnasium as gym
import panda_gym
from huggingface_sb3 import load_from_hub
from stable_baselines3 import SAC
from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize
repo_id = "monshinawatra/sac-her-PandaPickAndPlace-v3"
checkpoint = load_from_hub(repo_id, "sac-her-PandaPickAndPlace-v3.zip")
stats_path = load_from_hub(repo_id, "vec_normalize.pkl")
env = DummyVecEnv([lambda: gym.make("PandaPickAndPlace-v3", render_mode="human")])
env = VecNormalize.load(stats_path, env)
env.training = False
env.norm_reward = False
model = SAC.load(checkpoint, env=env)
obs = env.reset()
for _ in range(1000):
action, _ = model.predict(obs, deterministic=True)
obs, reward, done, info = env.step(action)
env.render()
Full inference script (with evaluation loop): inference.py.
Environment
- Env ID:
PandaPickAndPlace-v3 - Observation space:
Dict(observation,achieved_goal,desired_goal) — requiresMultiInputPolicy - Action space:
Box(4,)— 3D end-effector displacement + gripper - Max episode steps: 50