kp-deid-xlmr-560m-legal — a re-identification protector for legal text
A 560M XLM-RoBERTa-large token classifier (LoRA-finetuned, adapter merged) for PII de-identification, trained for the EuroPriv-Bench program. It is published not because it tops detection-F1 — it does not — but because, on the metric that measures privacy harm, it is the best protector we have found on real legal text.
Headline (real, external gold): on the Text Anonymization Benchmark (TAB; Pilán et al. 2022 — real ECHR court judgments, manually annotated), this model leaves 9.5% of DIRECT identifiers un-redacted vs the next-best public system's 49.6% (spaCy) / 50.0% (Presidio) — i.e. it leaks ~5× fewer of the identifiers that directly re-identify a person.
- Robust: 3 training seeds → DIRECT-leak {9.5%, 9.9%, 9.9%}.
- Significant: paired bootstrap over 264 shared DIRECT subjects → Δ vs runner-up = −40 points, 95% CI [−47.7, −32.2] (fully below 0).
- Not over-redaction: it marks 9.1% of tokens as PII (gold = 11.3% — it under-tags overall), with 4.3% false-positive redaction. It is aimed, not blunt.
Why detection-F1 is the wrong number, and re-id is the right one
This model sits mid-table on strict detection-F1 (~0.34). That is by design and is the whole point: strict F1 punishes disagreeing with the gold on a span's exact boundary or entity type — but a person's privacy does not care about the label, only about whether the identifying string was redacted. This model reliably touches the identifier tokens (removing them) even when it gets the boundary or type slightly wrong. If you choose a de-identifier by detection-F1, you can choose one that leaves half the identifiers behind. Choose by leak-rate.
Training data
Trained on real CJEU judgment structure (from davidwickerhf/cjeu-opendata, Apache-2.0) — real
legal document layouts — with synthetic, checksum-valid PII spliced in (gold by construction).
This is real legal structure with synthetic identifiers; it contains no real personal data.
CJEU (Court of Justice of the EU) is a different court from ECHR, so the TAB result above is genuine
zero-shot generalisation (contamination checked: 0 document overlap).
Taxonomy
18 GDPR-aligned KP entity types (BIOES): PERSON, NATIONAL_ID, ACCOUNT_ID, ADDRESS, EMAIL, PHONE, DATE, URL, ORG_PARTY, CASE_NUMBER, COURT, STATUTE_REF, COMPANY_ID, FACILITY, PROVIDER, MRN, HEALTH_CONDITION, SECRET.
Intended use & limitations
- Use: de-identification of English (legal) text where minimising re-identification risk is the priority — detect spans, then redact/replace.
- Scope of the claim: validated on one board (TAB, English legal). Breadth across languages and domains is in progress; treat use outside English legal as unvalidated.
- Research checkpoint (seed 0 of 3). Not clinical-validated. Strict-F1 is modest — expect boundary/type imperfection that does not affect redaction coverage.
- The re-id leak-rate metric is grounded in TAB's own externally-annotated DIRECT/QUASI labels (it is not a self-defined construct), but competitors report detection-F1, so head-to-head re-id numbers come from scoring all systems through EuroPriv-Bench.
Links
- Benchmark & leaderboard: research.klusai.com/leaderboard
- Write-up: Every detector leaked half the IDs on real court text — except one (research.klusai.com/blog)
- Comparand: klusai/kp-deid-mdeberta-280m
config_status: dev (synthetic training data) / evaluated on real-external-gold (TAB). Not a peer-reviewed claim yet — see the scorecard for the full method and caveats.