EU AI Act Risk Classifier
Fine-tuned DistilBERT model that classifies AI system descriptions into EU AI Act risk tiers.
Intended Use
Classifies a natural-language description of an AI system into one of five categories:
minimal_risklimited_riskhigh_riskprohibited_riskreview(flagged for human review — description is ambiguous or spans multiple tiers)
Out of scope: This model does not provide legal advice and should not be used as the sole basis for a compliance decision.
Training Data
- Dataset size: 1,538 labeled examples
- Labels sourced via a dual-labeling pipeline with human review on flagged conflicts
- Ambiguous/boundary-testing "review" cases which require human review.
Training Procedure
- Base model: distilbert-base-uncased
- Learning rate: 3e-5
- Epochs: 10 - 20
- Batch size: 8
- Max sequence length: 256
Evaluation Results
| Metric | Score |
|---|---|
| Accuracy | 92.08% |
| Macro F1 | 0.9209 |
Per-class F1:
| Class | F1 |
|---|---|
| minimal_risk | 0.97 |
| limited_risk | 0.93 |
| high_risk | 0.91 |
| prohibited_risk | 0.93 |
| review | 0.84 |
Limitations
- The "review" class has the lowest F1, the model is least confident distinguishing genuinely ambiguous cases, which is expected since these cases are excluded from training by design.
- High Risk and Limited Risk are the most frequently confused pair among the four primary tiers, particularly at the boundary between Annex III decision-influencing systems and Article 50 transparency-only systems.
Ethical Considerations / Risks
- The training labels came from an LLM-assisted first pass, then a decision tree based second pass, then a human reviewer.
- The results of the model are not intended to provide legal advice
Version History/Release Notes
v1.0 — 2026-08-14
- Initial release
- Trained on 1,538 labeled examples across 5 tiers: minimal_risk, limited_risk, high_risk, prohibited_risk, review
- Labels produced via a dual-pipeline: AI-assisted first pass → decision-tree second pass → human review, with conflicts between the two automated passes flagged for review
- Accuracy: 92.08% · Macro F1: 0.9209
- Per-class F1: minimal_risk 0.97, limited_risk 0.93, high_risk 0.91, prohibited_risk 0.93, review 0.84
- Known limitation: review has the lowest F1 of the five classes — expected, since ambiguous cases are excluded from confident training signal by design. High Risk / Limited Risk is the most frequently confused pair among the four primary tiers, particularly at the Annex III (decision-influencing) vs. Article 50 (transparency-only) boundary.