REINFORCE / actor-critic (CartPole)
A policy-gradient agent with a value baseline + entropy bonus that solves the Gymnasium CartPole-v1 benchmark (greedy return near the 500 maximum).
Trained from scratch in Ropedia Academy — an interactive, bilingual course on embodied & spatial AI. Educational model: small and quick to train; the value is the method and a reproducible pipeline, not a leaderboard score. Try it live in the Ropedia demos Space.
At a glance
| Base model | Trained from scratch (random initialization) — no pretrained base model. |
| Task | reinforcement learning |
| Training objective | REINFORCE policy gradient with a value baseline + entropy bonus (actor-critic). |
| Track | AG · Agents & RL |
| Notebook |  |
Dataset
- Name: Gymnasium CartPole-v1
- Type: standard RL benchmark environment (no fixed dataset)
- Size / stats: 4-D continuous state, 2 discrete actions; reward +1/step, episode cap 500; agent learns from its own rollouts
- Split: online RL; greedy eval over 20 episodes
- Source: Gymnasium (Farama Foundation) CartPole-v1
Training config
Actor-critic; Adam (lr 3e-3), 600 episodes, γ=0.99, entropy 0.01, value baseline. EPISODES env-overridable.
Evaluation results
| metric | value | meaning |
|---|---|---|
return (final) | 452.7 | |
greedy_eval | 449.45 | mean return of the greedy policy over 20 CartPole episodes (max 500) |
Robustness (mean ± std over 5 seeds)
Single-run numbers above are one seed; this is the distribution over independent re-trains (honest variance — no cherry-picking).
| metric | mean ± std |
|---|---|
greedy_eval | 461.8 ± 76 |
Inference example
import torch, torch.nn as nn
policy = nn.Sequential(nn.Linear(4,64), nn.Tanh(), nn.Linear(64,2))
policy.load_state_dict(torch.load("policy.pt", map_location="cpu")); policy.eval()
# CartPole state s = [x, x_dot, theta, theta_dot]
# action = int(policy(torch.tensor(s, dtype=torch.float32)).argmax()) # 0 = push left, 1 = push right
Limitations
Educational scale. Trained quickly on CPU on small or synthetic data, so absolute numbers are not competitive with production systems — the value is the method and a reproducible pipeline. No large-scale data, no hyperparameter sweep, and no multi-seed variance is reported. Not for production use.
Solves CartPole only — not a general control policy; on-policy and high-variance.
Failure cases
High-variance gradients; a length-1 episode gives a NaN advantage (std 0) — fixed with unbiased=False + grad clipping; stalls without the entropy bonus.
Reproduce / train your own
One click: open the notebook in Colab → Runtime → GPU → Run all, then run its Publish to the Hugging Face Hub cell.
From a shell:
git clone https://github.com/ChaoYue0307/ropedia-academy.git && cd ropedia-academy
pip install torch numpy matplotlib scikit-learn scikit-image gymnasium
jupyter nbconvert --to notebook --execute notebooks/training/AG_reinforce_gridworld.ipynb --output run.ipynb
# optional: override training length, e.g. STEPS=2000 (or EPISODES=600) before running
Files
figure.pngmetrics.jsonpolicy.ptseeds.png
License
Code & weights: MIT (this repository) — educational use encouraged.
Environment: Gymnasium (Farama Foundation) — MIT license.
Citation
If you use this model or the course materials, please cite:
@misc{ropedia_academy,
title = {Ropedia Academy: an interactive course on embodied & spatial AI},
author = {Ropedia Academy},
year = {2026},
howpublished = {\url{https://chaoyue0307.github.io/ropedia-academy/}}
}
Method / original work: Williams, REINFORCE, 1992; Sutton et al., Policy Gradient Methods, NeurIPS 1999; Brockman et al., OpenAI Gym, 2016.
Related assets
- 🚀 Live demos: https://huggingface.co/spaces/cy0307/ropedia-demos
- 🤗 All trained models + collection: https://huggingface.co/cy0307
- 📚 Course & all labs: https://chaoyue0307.github.io/ropedia-academy/ · Labs tab
- 💻 Source / notebooks: github.com/ChaoYue0307/ropedia-academy
Part of the Ropedia Academy trained-model collection. Contributions & issues welcome on GitHub.