Evaluation Metrics
The table below summarizes the detailed evaluation results per PII label:
| Label | TP | FP | FN | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|---|---|---|
| SURNAME | 3722 | 0 | 28 | 99.25% | 100.0% | 99.25% | 99.63% |
| O (Non-PII) | 0 | 400 | 0 | 99.30% | n/a | n/a | n/a |
| TIME | 1936 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% |
| DRIVERLICENSENUM | 505 | 0 | 2 | 99.61% | 100.0% | 99.61% | 99.80% |
| PASSPORTNUM | 564 | 0 | 2 | 99.65% | 100.0% | 99.65% | 99.82% |
| GIVENNAME | 7548 | 0 | 172 | 97.77% | 100.0% | 97.77% | 98.87% |
| TELEPHONENUM | 3641 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% |
| BUILDINGNUM | 407 | 0 | 19 | 95.54% | 100.0% | 95.54% | 97.72% |
| AGE | 168 | 0 | 1 | 99.41% | 100.0% | 99.41% | 99.70% |
| DATE | 2335 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% |
| CITY | 1672 | 0 | 130 | 92.79% | 100.0% | 92.79% | 96.26% |
| TITLE | 349 | 0 | 35 | 90.89% | 100.0% | 90.89% | 95.23% |
| IDCARDNUM | 1998 | 0 | 22 | 98.91% | 100.0% | 98.91% | 99.45% |
| GENDER | 121 | 0 | 0 | 100.0% | 100.0% | 100.0% | 100.0% |
| CREDITCARDNUMBER | 557 | 0 | 1 | 99.82% | 100.0% | 99.82% | 99.91% |
| SEX | 78 | 0 | 1 | 98.73% | 100.0% | 98.73% | 99.36% |
| STREET | 1368 | 0 | 19 | 98.63% | 100.0% | 98.63% | 99.31% |
| TAXNUM | 345 | 0 | 12 | 96.64% | 100.0% | 96.64% | 98.29% |
| 2606 | 0 | 2 | 99.92% | 100.0% | 99.92% | 99.96% | |
| SOCIALNUM | 411 | 0 | 11 | 97.39% | 100.0% | 97.39% | 98.68% |
| ZIPCODE | 406 | 0 | 20 | 95.31% | 100.0% | 95.31% | 97.60% |
Overall Evaluation
-
Accuracy: 99.01%
-
Precision: 98.72%
-
Recall: 98.47%
-
F1 Score: 98.59%
-
Total True Positives (TP): 30,737
-
Total False Positives (FP): 400
-
Total False Negatives (FN): 477
Macro-Averaged Metrics
- Accuracy: 98.35%
- Precision: 95.24%
- Recall: 93.35%
- F1 Score: 94.29%
Model Behavior & Limitations
- Evaluation Focus:
The metrics shown above reflect performance on the test split of the open-pii-masking-500k-ai4privacy dataset. Real-world performance may vary and requires additional measures. Feel free to contact support@ai4privacy.com for assistance.
Disclaimer
This model card details the evaluation metrics and fine-tuning parameters for the multilingual anonymiser. Please note:
- The model is provided as-is under the MIT License.
- It is intended solely for redaction purposes and does not perform full PII classification.
- Users should carefully test and evaluate its performance on their own data before deploying in production environments.
Ai4Privacy – Committed to protecting personal data in the age of AI.