Reinforce Agent playing Pixelcopter-PLE-v0
This is a trained model of a Reinforce (Monte-Carlo policy gradient) agent playing Pixelcopter-PLE-v0 (a Flappy-Bird-style helicopter game from the PyGame Learning Environment). It was trained with a custom-from-scratch REINFORCE implementation (not Stable-Baselines3) as part of Unit 4 of the Hugging Face Deep RL Course.
The environment
The agent flies a helicopter through a scrolling cavern, avoiding the ceiling, the floor, and block obstacles. Each step it chooses to flap up or do nothing; it earns a small positive reward for progressing and a negative reward on crash. Episodes are long-horizon and rewards are sparse/noisy, which makes vanilla REINFORCE notably harder to train here than on CartPole.
- Observation space (7): player y-position and velocity, plus distances/positions to the next gate and cavern walls
- Action space (2): flap up / do nothing
Results
| Metric | Value |
|---|---|
| Mean reward (10 eval episodes) | 73.00 +/- 47.63 |
| Best training rolling-avg (100-ep) | 61.28 @ episode 46617 |
The high std reflects Pixelcopter's noisy, long-horizon dynamics — some runs go far, others crash early.
Policy architecture
An MLP outputting a categorical action distribution:
Linear(7 -> 64) -> ReLU -> Linear(64 -> 128) -> ReLU -> Linear(128 -> 2) -> Softmax
Training hyperparameters
| Hyperparameter | Value |
|---|---|
env_id | Pixelcopter-PLE-v0 |
h_size | 64 |
n_training_episodes | 50000 |
n_evaluation_episodes | 10 |
max_t | 10000 |
gamma | 0.99 |
lr | 0.0001 |
| optimizer | Adam |
Note: this legacy PLE/gym_pygame environment uses the old Gym step API, so training wraps it
with gym.make(..., apply_api_compatibility=True).
Usage
The model is a pickled PyTorch policy (model.pt). Loading it requires the PixelCopterPolicy
class from this repo's training code (policies.py), plus gym_pygame installed.
import torch
from huggingface_hub import hf_hub_download
from policies import PixelCopterPolicy # from the Unit 4 training code
model_path = hf_hub_download(repo_id="kaleido-jean/Reinforce-Pixelcopter-1", filename="model.pt")
policy = torch.load(model_path, weights_only=False)
policy.eval()
import gym, gym_pygame # noqa: F401
env = gym.make("Pixelcopter-PLE-v0", apply_api_compatibility=True)
state, _ = env.reset()
done = False
while not done:
action, _ = policy.act(state, "cpu")
state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
Reproduce
python train.py --task pixelcopter --hf
To learn to use this model and train your own, check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction