Reinforce Agent playing CartPole-v1
This is a trained model of a Reinforce (Monte-Carlo policy gradient) agent playing CartPole-v1. It was trained with a custom-from-scratch REINFORCE implementation (not Stable-Baselines3) as part of Unit 4 of the Hugging Face Deep RL Course.
The environment
CartPole-v1 is the classic pole-balancing control task: a pole is attached by an un-actuated joint to a cart that moves along a frictionless track. The agent applies a force (left / right) each step and earns +1 reward per timestep the pole stays upright. An episode ends when the pole falls past ±12°, the cart leaves the track, or 500 steps are reached — so 500 is the maximum achievable return.
- Observation space (4): cart position, cart velocity, pole angle, pole angular velocity
- Action space (2): push cart left / right
Results
| Metric | Value |
|---|---|
| Mean reward (10 eval episodes) | 500.00 +/- 0.00 |
| Max possible return | 500 |
The agent reaches and holds the optimal policy (perfect balance for the full episode length).
Policy architecture
A small MLP outputting a categorical action distribution:
Linear(4 -> 16) -> ReLU -> Linear(16 -> 2) -> Softmax
Training hyperparameters
| Hyperparameter | Value |
|---|---|
env_id | CartPole-v1 |
h_size | 16 |
n_training_episodes | 1000 |
n_evaluation_episodes | 10 |
max_t | 1000 |
gamma | 0.99 |
lr | 0.01 |
| optimizer | Adam |
Usage
The model is a pickled PyTorch policy (model.pt). Loading it requires the CartPolePolicy
class from this repo's training code (policies.py).
import torch
from huggingface_hub import hf_hub_download
from policies import CartPolePolicy # from the Unit 4 training code
model_path = hf_hub_download(repo_id="kaleido-jean/Reinforce-Cartpole-1", filename="model.pt")
policy = torch.load(model_path, weights_only=False)
policy.eval()
import gym
env = gym.make("CartPole-v1")
state, _ = env.reset()
done = False
while not done:
action, _ = policy.act(state, "cpu")
state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
Reproduce
python train.py --task cartpole --hf
To learn to use this model and train your own, check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction