modernbert-pii-v0.1
ModernBERT-base fine-tuned for PII/PHI span detection over a 169-label BIO space derived from a versioned 84-concept ontology.
Training
| base model | answerdotai/ModernBERT-base |
| objective | scheme-aware marginal loss (not plain cross-entropy) |
| steps | 33,100 |
| max length | 512 |
| label level | core (ontology v0.1) |
| seed | 0 |
Scheme-aware loss
The 32 source corpora annotate different label sets. A corpus that
annotates 7 concepts tags every email and phone number as O — under plain
cross-entropy those become hard negatives and the model unlearns them. Instead
each token is trained against the set of labels consistent with its
annotation: for O, that is O plus every concept the source dataset does not
annotate.
Training datasets (32)
ai4privacy/legacy-tiersai4privacy/open-pii-500kai4privacy/openpii-1.5malrosait/russian-piibccard/openpii-financebob14/korean-sensitivebtx24/turkish-piicareons/dutch-healthcare-piidatafog/test-corpusdataiku/kijidialogpiie3jsi/multi-pii-nerfelipe53/name-judgmentgrascco/multigrascco/pii-v2gretel/pii-finance-multilingualgretel/pii-masking-enkdpiikeler/turkish-medical-deid-evalkierandesmond/gdpr-special-categorieslacuna/piilocaldoc/azerbaijani-piimeddocanmeddoplacemeddoprofnisaefendiolu/code-secretsnutrient/docpiinvidia/nemotron-piipresidio/research-evalsidharth/indicphitab/text-anonymization-benchmarktonicai/privacy-bench
Known limitations
- Character offsets from the tokenizer include the preceding space; decoded spans must be whitespace-trimmed. Without it, span-exact accuracy falls from ~99.7% to ~5.9% while token-level F1 is unchanged.
- Spans located by string search rather than given by the annotator are marked
derivedin the corpus; they train but were never scored. - Coverage is skewed: health identifiers are a small fraction of training mentions relative to network and contact identifiers.