DQN — ALE/SpaceInvaders-v5
A from-scratch PyTorch reimplementation of DeepMind's DQN Atari papers (Mnih et al.,
2013, Mnih et al.,
2015), trained on ALE/SpaceInvaders-v5.
- Code / training pipeline: https://github.com/colichar/deep-q-learning
- Architecture: DeepMind's original conv/FC Q-network (3 conv layers + 2 FC layers), operating on stacks of 4 grayscale 84x84 frames.
- Framework: PyTorch.
Training setup
Trained for 50M frames (~12.5 hours) on a single RTX 4070 Ti Super, with a single gymnasium environment
(num_envs=1). Hyperparameters match the DeepMind 2015 paper, with a centered RMSprop optimizer:
| Hyperparameter | Value |
|---|---|
| Optimizer | RMSprop (centered, alpha=0.95, eps=0.01) |
| Learning rate | 0.00025 |
| Discount (gamma) | 0.99 |
| Replay memory size | 1,000,000 frames |
| Memory warmup | 50,000 frames |
| Batch size | 32 |
| Target network sync | every 10,000 frames |
| Main network update | every 4 frames |
| Frame skip | 4 (with max-pooling over the last 2 frames for flicker removal) |
| Reward clipping | {-1, 0, 1} |
See the repo README for the full flag list.
Results
Over the last 1M frames of training (501 episodes): average episode reward ≈ 1968.6, max episode reward 4155.
Reward is the 200-episode rolling mean (shaded band: raw per-episode reward); loss is the Huber loss averaged every 400 frames, on a log scale.
Files
model.pth—torch.savedict withmodel_state_dictandoptimizer_state_dictfor the main Q-network.episodes.csv— per-episodeframe_num,episode_num,episode_reward,epsilon,wall_clock_elapsed_seconds, full training run.losses.csv—frame_num,avg_loss, Huber loss averaged every 400 frames.training-curves.png— the plot above, generated from the two CSVs.
The replay memory buffer (~14GB) used to resume training locally is intentionally not included here — it's only needed to resume training, not to run or evaluate the model.
Usage
The checkpoint's state dict matches src.models.cnn_model_py.CNNModelPY in the linked repo and is loaded through
SpaceInvaderAgent, not standalone torch.load + a bare nn.Module — see
scripts/evaluate.py:
git clone https://github.com/colichar/deep-q-learning
cd deep-q-learning
uv sync --extra cpu # or --extra cuda
uv run AutoROM --accept-license -y
# download this checkpoint's model/ dir from the Hub into, e.g., checkpoint/model/
uv run python scripts/evaluate.py --checkpoint checkpoint --optimizer rmsprop --episodes 5
Limitations
- Trained and evaluated on a single game (
ALE/SpaceInvaders-v5); the network is not general-purpose across Atari games without retraining. - Intended for research/education (reproducing the DQN papers), not as a production game-playing agent.
License
MIT — see LICENSE in the linked repo.
Citation
@article{mnih2013playing,
title={Playing atari with deep reinforcement learning},
author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin},
journal={arXiv preprint arXiv:1312.5602},
year={2013}
}
@article{mnih2015human,
title={Human-level control through deep reinforcement learning},
author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others},
journal={Nature},
volume={518},
number={7540},
pages={529--533},
year={2015},
publisher={Nature Publishing Group}
}