Rocket League mistake detection models
Versioned models for identifying potential player mistakes in Rocket League replays. This repository stores releases for different mistake kinds, with separate versions and evaluation reports for each model.
The rl-mistake-analysis project identifies candidate incidents, extracts features, and runs inference. Replay-analysis applications can use the predictions for review and coaching.
A candidate is an incident worth reviewing. Features are measurements describing it. A model uses those measurements to estimate whether it is a mistake. Different mistake kinds can use different models.
Model catalog
The root manifest.json lists available mistake kinds and model versions. Each release has its own directory:
manifest.json
models/
<kind>/
<version>/
model.onnx
evaluation.json
The manifest records feature contracts, thresholds, evaluation paths, checksums, source provenance, and any separate ONNX weight files. New versions get new directories; released paths stay stable. Catalog and model versions are independent. Downstream applications can choose which release to activate.
Using a model
Choose a mistake kind and version from the catalog. With the Hugging Face CLI installed, download that release:
hf download alina-afanaseva/rl-mistake-analysis-models \
--include "manifest.json" \
--include "models/<kind>/<version>/*" \
--local-dir "/path/to/model-repository"
Replace <kind>, <version>, and /path/to/model-repository with your chosen release and destination. This downloads the catalog and all files for that release. Pass the directory, kind, and version to the rl-mistake-analysis inference library. It uses ONNX Runtime and the release's decision threshold.
Mistake kinds
Teammate bumps (bumping_teammate)
This mistake kind uses the play around a teammate collision to judge it from each player's perspective. Each participant is scored separately. Being the physical initiator does not automatically make a player responsible.
Features describe motion, positioning, ball touches, possession, and surrounding events. See the teammate-bump documentation for the mistake definition and feature details.
Training and evaluation
Human-reviewed collisions provide mistake and non-mistake labels for individual players. Ten classifier families are compared: CatBoost, XGBoost, Random Forest, Extra Trees, an Explainable Boosting Machine, two logistic regression variants, and linear, RBF, and polynomial SVMs.
Cross-validation keeps whole replays together, including both collision participants. Preprocessing is learned within training folds; model settings and thresholds are selected by mean validation F1. Reserved test replays do not select parameters or thresholds.
Precision measures how many flags are correct; recall measures how many labeled mistakes are found. F1 balances the two. Each release's evaluation.json records its scores and evaluation setup. DVC preserves data snapshots, and MLflow records experiments. ONNX probabilities are checked against the original models on training and test rows.
Release 0.1.0
Release 0.1.0 uses unweighted XGBoost with 90 features, selected by the highest mean validation F1 among ten classifier families. The frozen dataset contains 2,108 player-specific judgments from 367 replays: 1,838 records from 304 replays for training and cross-validation, and 270 records from 63 reserved test replays. Validation scores are the mean of three stratified, replay-grouped folds with seed 42. Training scores use the final model fitted on all training records. Each family's selected probability threshold was 0.40.
All scores in this table are percentages. Average precision summarizes the precision-recall curve.
| Model | Train F1 | Validation F1 | Test F1 | Test precision | Test recall | Test ROC AUC | Test average precision |
|---|---|---|---|---|---|---|---|
| XGBoost | 71.50 | 69.30 | 73.22 | 67.68 | 79.76 | 88.68 | 81.79 |
| Random Forest | 75.83 | 68.99 | 74.07 | 66.67 | 83.33 | 88.58 | 82.18 |
| CatBoost | 78.29 | 68.92 | 73.91 | 68.00 | 80.95 | 88.12 | 79.64 |
| Additive EBM | 73.07 | 67.86 | 69.23 | 64.29 | 75.00 | 86.49 | 75.75 |
| Extra Trees | 77.82 | 66.73 | 72.34 | 65.38 | 80.95 | 86.25 | 78.45 |
| Elastic-net logistic | 70.31 | 66.50 | 68.06 | 60.75 | 77.38 | 85.35 | 72.84 |
| SVM polynomial | 76.22 | 65.59 | 67.03 | 62.24 | 72.62 | 85.82 | 72.57 |
| Logistic regression | 69.79 | 65.27 | 68.82 | 62.75 | 76.19 | 84.75 | 71.17 |
| SVM RBF | 81.71 | 65.21 | 70.00 | 65.63 | 75.00 | 87.12 | 75.47 |
| SVM linear | 71.07 | 64.01 | 69.27 | 65.26 | 73.81 | 84.18 | 68.72 |
We also evaluated all 105 features. The additional 15 included redundant context measurements and did not provide a consistent improvement, so the baseline retains 90 features.
The labels are imbalanced: approximately 35% mistakes and 65% non-mistakes. We tested mild and full inverse-frequency class weighting alongside unweighted training, deriving weights separately inside each training fold and tuning model parameters and thresholds. Balancing did not improve the metrics meaningfully or consistently. The published model uses no class weighting.
To check sensitivity to the CV seed, we evaluated fixed selected models on five additional replay-grouped split seeds, keeping their parameters, thresholds, model randomness and test replays unchanged. The baseline XGBoost's validation F1 ranged from 68.01% to 68.75%, with a mean of 68.47%. Its performance remained similar across these splits. The original seed was used when selecting the parameters and threshold, so its higher score is expected.
Limitations
- This is retrospective replay analysis: some features use events after the collision, so the models require that later context.
- All labels come from one human annotator and reflect that person's judgment.
- Scores support reviewing detected teammate collisions. They do not establish causation or cover other mistake types.
- Replay data and annotations are not distributed here, which limits independent reproduction of the reported scores.
License
The model artifacts and documentation in this repository are released under the MIT License.