PPO Policy — MuJoCo Reacher-v5 (rl-mujoco-reacher-ppo)
A PPO (stable-baselines3) actor trained to swing a 2-DOF planar robotic
arm so its fingertip reaches a randomly placed target — the canonical
continuous-control manipulation benchmark (gymnasium MuJoCo Reacher-v5).
Environment
- Observation (10-dim): cos/sin of both joint angles, target xy position, joint angular velocities, fingertip-to-target vector.
- Action (2-dim, continuous): joint torques in [-1, 1].
- Reward (dense):
-||fingertip - target|| - control_cost * sum(a^2); 50 steps per episode.
Results (50 deterministic eval episodes, fixed seeds)
| Policy | Mean episode return | Success rate (final dist < 0.05 m) |
|---|---|---|
| PPO (this model) | -7.66 ± 2.41 | 8% |
| Random baseline | -42.49 ± 3.67 | 4% |
Mean final fingertip-to-target distance: 0.1324 m; reaching diagnostic (fingertip within 0.05 m at some point in the episode): 62% of episodes.
Usage
model.py is dependency-light (torch + numpy + gymnasium only — no
stable-baselines3 needed):
from model import load_policy, rollout
policy = load_policy("pytorch_policy.bin") # or "ppo_reacher.zip" (needs SB3)
frames, episode_return, final_distance = rollout(policy, seed=0)
print(episode_return, final_distance)
The deterministic action is the Gaussian mean of the SB3 actor clipped to
[-1, 1] — exactly SB3 predict(deterministic=True). No observation
normalization was used in training.
Training details
- PPO: lr 3e-4, n_steps 2048 x4 envs, batch 64, 10 epochs, gamma 0.99, GAE lambda 0.95, clip 0.2; MLP [64, 64] tanh; 200,000 timesteps on CPU, seed 0.
- Full reproducible pipeline (training, evaluation, rollout rendering): see the project repository.
Limitations
- Reacher is a toy 2-DOF planar arm with a fully observed state and a hand-shaped dense reward — not contact-rich or partially observed manipulation.
- Simulation only: no sim-to-real claim; MuJoCo actuator/contact modelling gaps would dominate on hardware.
Built as a portfolio project for graduate applications in intelligent robotics / manipulation (mechanical engineering x AI).