AfriMed-PII-AfriBERTa: clinical PII de-identification for Luganda, Kiswahili, Kinyarwanda and Dholuo

AfriBERTa (Ogueji, Zhu and Lin, 2021; University of Waterloo, released through the Castorini group) is an encoder pretrained from scratch on under 1 GB of text in 11 African languages, including Kiswahili and Kinyarwanda (as Gahuza) but not Luganda or Dholuo. Of the three backbones released in this series it is the smallest and, model-for-model, has the highest patient-name recall in every language; its precision is the lowest of the three, so it finds the most identifiers and marks the most non-identifier text in doing so. The tables below quantify both.

This release contains four per-language checkpoints fine-tuned from castorini/afriberta_large for token-level detection of personal information in clinical text. Each checkpoint emits character-offset spans with one of 14 categories. The model outputs positions and labels; it does not generate text.

Model details

Checkpoints in this release

Each checkpoint is the seed selected on the development split (see Model selection). The remaining seeds are retained and are available on request; every figure below is a mean over all five.

Dholuo was included as a deliberate stress case. The other three languages are the jurisdiction-aligned core; Dholuo is Nilotic where they are Bantu, has one to two orders of magnitude less monolingual text (about 5 to 6 million tokens of licence-clean text in total), and no backbone in this series was pretrained on it. Its results therefore bound what the approach can do for a genuinely low-resource language of a different family, rather than measuring it under favourable conditions.

Categories

The 14 categories follow the project's medical-PII taxonomy (D3). Identifier categories with a fixed surface form, national ID, phone, date, record number, age, are detectable by pattern; the remaining categories require the model.

Uses

The model is a component for de-identifying clinical free text in the four languages. Its measured operating envelope is given in Evaluation; the per-category tables there describe what proportion of each kind of identifier it finds and how much non-identifier text it marks in doing so, and are the basis on which to judge fitness for a given workflow.

It performs best on clinical-register text. Its behaviour on other registers was measured on news text and is reported under Out-of-domain.

How to use

from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "CraneAILabs/afrimed-pii-afriberta"
lang = "lug"   # or "swa", "kin", "luo"

tok = AutoTokenizer.from_pretrained(repo, subfolder=lang)
mdl = AutoModelForTokenClassification.from_pretrained(repo, subfolder=lang)

The models were trained over a whitespace-and-punctuation word grid and expect input pre-split into words (tokenizer(words, is_split_into_words=True)), with predictions aligned back to character offsets. A reference implementation of that alignment, together with an optional rule layer for the fixed-format identifiers, is provided in deid.py in this repository. The three calls above were verified to load and run for all four language checkpoints on transformers 5.13 / torch 2.13 with no trust_remote_code and no additional files; warm inference on a 219-character record takes ~6 ms on a GB10 GPU and ~100 ms on CPU.

The system on a clinical note

The screenshots below are the deployed system (AfriBERTa checkpoints, rule layer and calibration) on a short synthetic clinical note per language, captured from the live demo. Category labels replace the spans the system found. They are shown unedited, including what the system gets wrong, because that is what a user of these models will see.

In the Luganda note the system finds the patient's name, the age, the national ID, the phone number, the date, both places and the father's name; nothing identifying is left in the text. Across the four notes the rule layer takes the age, the national ID, the phone number and the date in every language, the template layer takes the named facility in Kinyarwanda, Kiswahili and Dholuo, and the model takes the names and places. Two errors are visible and both are the model's: in the Dholuo note the district Nyando is typed as a third-party name (the location/person confusion reported under Bias, risks and limitations), and the patient-name span swallows the preceding word Jatuo (patient). Both still redact the identifier; the cost is a wrong label and one extra word.

Two properties of the rule layer matter when reading these notes. Age is matched in both word orders (46 emyaka and emyaka 46); the over-matches this produces are durations such as after 10 years of follow-up, which are not identifiers and are accepted as the cost of recall 1.000. Facility names in natural phrasing (Kiswahili Hospitali ya, Zahanati ya, Kituo cha Afya; Kinyarwanda Ibitaro bya, Ikigo nderabuzima cya; Dholuo osipital mar; Luganda eddwaliro lya) are labelled by the template layer. The translated test writes facility names in their English form, so these phrases never fire there (0 matches in 707 documents): the system figures in this card do not depend on them, and their recall on natural text is a claim from the naming conventions, not a measurement.

The full-page captures for all four languages, with the exact input and output text, are in figures/demo/.

Training details

Training data

The models were fine-tuned on a synthetic clinical corpus (internal designation D5, version 1): 8,000 training and 1,000 development documents per language, 36,000 in total. Documents are clinical-style records generated from language-specific templates, filled from curated East African vocabularies of given and family names, villages, districts and health facilities, and country-specific identifier formats for Uganda, Kenya, Rwanda and Tanzania. A native-speaker review of 600 documents was conducted across the four languages, and the Kinyarwanda partition was regenerated with reviewer-sourced lexical terms. Every span is labelled with one of the 14 categories above.

For this backbone the corpus was additionally regenerated so that age spans include their unit word (e.g. 46 emyaka, 46 miaka), matching the annotation convention of the evaluation corpus; the original convention labelled the bare number. This is the only change from the corpus used for the other two backbones and it accounts for the age-category results.

No real patient data was used. Evaluation corpora, including MasakhaNER, which is licensed for non-commercial use, were used for evaluation and development only and never for training. The training corpus itself is not part of this release.

Kinyarwanda tribe and clan are out of scope, by ruling. All three backbones score 0.000 on Kinyarwanda TRIBE and PERSON_CLAN on the synthetic evaluation set. This is not a detection failure: the project's taxonomy ruling removed both categories for Kinyarwanda, Rwandan clinical records do not record Hutu/Tutsi/Twa under post-1994 law and policy, and clan has a Luganda-register basis, so the Kinyarwanda training corpus the released weights learned from contains no spans of either. The evaluation edition reported here is an earlier one that still annotates 76 tribe and 35 clan spans; the zeros record that the released Kinyarwanda checkpoint was never trained to emit those labels, which is the intended behaviour. Kinyarwanda RELIGION, which remains in scope, scores 0.86 to 0.93.

Training procedure

Each language is trained separately as a token-classification model: the released base encoder with a randomly initialised linear head over 29 BIO tags, trained with cross-entropy over a whitespace-and-punctuation word grid (first sub-token of each word carries the label). The base weights are the public castorini/afriberta_large checkpoint; no continued pretraining was applied to this backbone, because in a controlled comparison it did not measurably help an encoder already pretrained on African-language text. Five seeds were trained per language.

Hyperparameters

Model selection

The learning rate was swept per language over {1e-5, 2e-5, 3e-5, 5e-5}, and the (learning rate, epoch) pair was chosen on a real held-out development set, the MasakhaNER validation split plus the 30% development slice of the translated-clinical corpus, by per-category macro-recall over categories with at least ten gold spans (age excluded as an annotation-convention artefact at the time). The synthetic development split, on which all configurations saturate near F1 0.99, was deliberately not used for selection because it does not predict performance on real text. The test split was never used for any selection. The released checkpoint per language is the seed selected on that development set; all five seeds are reported.

Compute

Fine-tuning and the learning-rate sweep ran on a single NVIDIA DGX Spark (GB10). The complete lane for this backbone, the learning-rate sweep, two ablated levers, the AGE-convention regeneration and five-seed finals for four languages, with scoring, used 7.0 GPU-hours as logged by the lane; 5-seed training of the four released configurations alone accounts for roughly 1.3 of those.

Evaluation

Evaluation data

Three evaluation sets are reported. Together they cover every category and separate three questions, performance on the clinical register, generalisation to names never seen in training, and behaviour out of domain.

1. Held-out translated-clinical test (teval). The MEDDOCAN Spanish clinical corpus (Marimon et al., 2019) machine-translated into each of the four languages with span offsets re-aligned; split 30% development / 70% test by document index. Test size per language: Luganda 161 docs / 3,351 spans; Kiswahili 163 / 3,387; Kinyarwanda 158 / 3,278; Dholuo 168 / 3,487. The source schema has no tribe, religion or clan annotations, so those categories are evaluated on set 3.
2. Name-disjoint variant (teval-novel). The same test documents with every person-name span replaced by a well-formed surrogate drawn from a Wikidata (CC0) name pool whose tokens are disjoint from the training vocabulary, verified at 0.0% overlap in all four languages against nine sources, with document and span counts preserved exactly. A reshuffle control (original names permuted across documents) separates the effect of unfamiliar names from that of merely different names.
3. Synthetic development set (d5-dev), an in-distribution ceiling, not a held-out score. 1,000 documents per language from the synthetic generator. These are the documents on which training early-stopped (best epoch selected by F1 on this set), so no training document appears in it but the released weights are, by construction, the epoch that scored best here. Its figures are therefore an upper bound for in-distribution synthetic text. It is the only set carrying gold spans for TRIBE, RELIGION and PERSON_CLAN, and is reported for those categories only, labelled as such.

Out-of-domain behaviour is measured on the MasakhaNER 1.0 test split (news text; PERSON, LOCATION, DATE), which is used for evaluation only.

Metrics

Scores are entity-level, strict: a predicted span counts only if its character offsets and category both match the gold span exactly. A relaxed score (any character overlap with matching category) is also reported where it was computed, as a diagnostic of boundary behaviour: the gap between the two is the share of identifiers the model finds but mis-delimits. All figures are micro averages unless labelled per category, and are means over five training seeds with the seed standard deviation.

We report precision, recall and F1 together, because in de-identification each answers a different question, and this follows the convention of the field's shared tasks. The 2014 i2b2/UTHealth track reported entity-level micro precision, recall and F1 with F1 as the primary metric for comparing systems; MEDDOCAN 2019 reported the same and additionally a leak score, the proportion of identifiers missed, as an explicit safety measure, observing that systems tend to favour precision over recall. Accordingly:

• Recall is the safety metric: an identifier the model does not find remains in the text.
• Precision is the utility metric: text wrongly marked as an identifier is removed unnecessarily.
• F1 is the comparison metric between systems.
• Leak = 1 − recall, reported per category. This is MEDDOCAN's safety measure and is included so the tables are read in the same terms as that benchmark.

Strict versus relaxed, and why both appear. Under the strict criterion a prediction counts only if its character offsets and its category both match the gold span exactly; this is the criterion the shared tasks use and it is the headline everywhere in this card. Under the relaxed criterion a gold span counts as found if some prediction of the same category overlaps it by at least one character. Relaxed is not an alternative score and is never compared against strict figures from elsewhere; it is a diagnostic that separates two different failures. When strict is low and relaxed is high, the model found the identifier but drew its boundary differently (an attached determiner, included punctuation, a tokeniser split); when both are low, it did not find it. A relaxed value of 1.000 therefore means every gold span of that category was touched by a correctly-typed prediction, which is common for well-formed categories such as dates and is the reason those values are not rare here. The clearest case in this series is age on the XLM-R checkpoints: strict 0.00, relaxed 0.98 to 0.99, because the model tags the number and the gold span includes the unit word.

Seed-to-seed variation in name recall spans SD 0.008-0.094 across languages; per-language differences smaller than about 0.05 are within that variation and are not interpreted.

Results, held-out translated-clinical test (model only)

These tables describe the released weights alone: no rule layer and no threshold adjustment. Each row gives the number of gold spans, so low-count categories can be weighed accordingly.

Luganda, held-out translated-clinical test, model only, 5 seeds

Not in this table: tribe, religion, clan. The translated test carries no gold spans for them (the source schema has none); they are scored on the synthetic development set under Results, tribe, religion and clan below.

Kiswahili, held-out translated-clinical test, model only, 5 seeds

Not in this table: tribe, religion, clan. The translated test carries no gold spans for them (the source schema has none); they are scored on the synthetic development set under Results, tribe, religion and clan below.

Kinyarwanda, held-out translated-clinical test, model only, 5 seeds

Not in this table: tribe, religion, clan. The translated test carries no gold spans for them (the source schema has none); they are scored on the synthetic development set under Results, tribe, religion and clan below.

Dholuo, held-out translated-clinical test, model only, 5 seeds

Not in this table: tribe, religion, clan. The translated test carries no gold spans for them (the source schema has none); they are scored on the synthetic development set under Results, tribe, religion and clan below.

Across the three released backbones

The same fine-tuning data, recipe family and evaluation were used for all three backbones in this series, so the figures below are directly comparable. Patient-name recall is the safety-critical category; micro F1 is the balanced comparison metric.

Results, names never seen in training

Δ is recall on unseen names minus recall on familiar names. The reshuffle control keeps the original names but permutes them across documents, so it measures context change alone; the part of Δ not explained by the reshuffle column is the effect of name novelty itself. Model only, entity-strict.

The deployed system on the same unseen-name test. The name thresholds were fitted on the unseen-name development slice with a no-recall-regression guard. On the test slice it holds only approximately: the system gives back between 0.006 and 0.010 of patient-name recall on unseen names, against micro F1 gains of 0.20 to 0.42. Held-out unseen-name test, entity-strict, 5-seed mean.

Results, tribe, religion and clan (synthetic dev; in-distribution ceiling)

Synthetic development set, the set training early-stopped on, so an in-distribution ceiling rather than a held-out score. Reported because it is the only set with gold for these categories, which the clinical test corpus lacks. Kinyarwanda tribe and clan are out of scope by ruling (see Training data).

All three released backbones on the same synthetic development set; AfriBERTa is a 5-seed mean, XLM-R and AfroXLMR a single checkpoint. Kinyarwanda tribe and clan are out of scope by ruling and carry no bar.

Out-of-domain

MasakhaNER 1.0 test (news). Names in news text are drawn from a different distribution and register than clinical notes; the gap between this table and the clinical results is the model's domain sensitivity.

Comparison with off-the-shelf systems

Publicly available de-identification and NER systems were run zero-shot on the same held-out test split with the same scorer, so the comparison is like for like. None was trained for these languages or this task; they are the systems a practitioner could pick up today, and the comparison measures what is available without training, not what each architecture could reach if fine-tuned on this data.

GLiNER-multi deserves a specific note, because its zero-shot label prompts can be chosen and its confidence threshold can be set, and a fixed choice understates it. It is shown twice: at its published default (the 13 category names as prompts, threshold 0.3), and at the best operating point reachable by tuning the prompt wording and threshold on the development split, read once on the test split. Tuning lifts it by 0.04 to 0.12 F1, almost entirely from clinical-descriptive prompts and a lower threshold. It remains a general-domain model trained on English-centric span data with no clinical text and none of these four languages, so it is being asked to transfer across domain and language at once. On Kinyarwanda the tuned GLiNER exceeds the model-only score of every released backbone; the released system with its rule layer and calibration (next section) is well above it in every language.

Micro F1, entity-strict, on the same held-out translated-clinical test. The off-the-shelf systems are run zero-shot; this model is fine-tuned on synthetic data for the task. The tuned GLiNER row selects, per language, the label-prompt set and confidence threshold with the best micro F1 on the development split only, then reads the test split once (Luganda V2 at 0.25, Kiswahili V3 at 0.15, Kinyarwanda V2 at 0.1, Dholuo V2 at 0.15; V2 is a clinical-descriptive prompt set, V3 adds a generic person label; the full grid is in baselines_gliner_calibrated.json and GLINER_CALIBRATION.md).

Effect of a rule layer and threshold calibration (not part of these weights)

The deployable system built on these weights adds two components that are not part of the released model and are reported separately so the model's own performance is not conflated with them.

Why a rule layer. National ID, phone, date, record number, other contact and age have fixed surface forms. Deterministic detectors handle them, and the model is used for names and places. Measured on the held-out translated-clinical test (70% slice), rule layer alone, entity-strict: recall 1.000 in every category in all four languages; precision 1.000 for national ID, record number and phone, 0.987 to 0.994 for other contact, 0.997 to 0.998 for date, and 0.894 to 0.913 for age. The age precision is the cost of a deliberate choice: the age rule accepts both word orders (46 emyaka and emyaka 46), and in this corpus the unit-first form also appears in durations such as emyaka 3 ("three years", of treatment or follow-up), which the rule redacts and the gold does not mark. That is an over-redaction of a number that is not an identifier, accepted so that no patient age is left in the text. Record numbers that carry a facility's own prefix rather than a register name (MUL/2024/00871, KNH/24/00871) are covered by a structure rule (two to four capitals, a two- or four-digit year, four to six digits): it never fires on the translated test, where the register rule already reaches recall 1.000 on 2,078 gold spans, and the same structure matches 1,109 times there, all inside gold.

Why calibration on unseen names. The model attaches a confidence to each span. Fitting the decision threshold on the ordinary development split, where names are token-familiar and confidence is high, suppresses correct low-confidence predictions on unfamiliar names. The name thresholds are therefore fitted on the name-disjoint development slice, with an asymmetric precision floor and a no-recall-regression guard. The same procedure, unchanged, was applied to all three backbones.

Same test split, entity-strict; supervisor re-scored. The effect has the same shape on all three backbones: the gain is predominantly precision, with recall preserved or improved.

How the calibration is done. The model attaches a confidence to every span it proposes. A single decision threshold is applied per language: spans above it are kept, spans below it are dropped. The threshold is fitted, never hand-set, on development data only, in three steps. First, a rule layer takes over the fixed-format categories (national ID, phone, date, record number, age), where deterministic detectors reach recall 1.000, so the model's threshold only governs names and places. Second, the threshold for the two name categories is fitted on the name-disjoint development slice, the same documents used for the unseen-name results, rather than on the ordinary development split. This matters because on familiar names the model is confident and any threshold looks fine; on unfamiliar names its confidence is lower, and a threshold fitted on familiar names silently discards correct low-confidence predictions. Fitting on unfamiliar names recovered most of that recall (for example, unseen-name patient-name recall on the AfriBERTa checkpoints rose from 0.343 to 0.494 in Luganda and from 0.235 to 0.524 in Dholuo) at a cost of 1 to 2 percentage points of over-redaction. Third, the fit is asymmetric: for names it maximises recall subject to a precision floor, because a missed name is a disclosure while an over-redacted token is only lost text, and a guard rejects any threshold that would lower recall relative to the uncalibrated model. The test split is not read at any point in this procedure. The identical procedure was applied to all three backbones; the shipped thresholds are in router_config.json.

Evaluation on real clinical records

These models have not been evaluated on real clinical records in any of the four languages. No annotated corpus of that kind existed when this work was done; the clinical evaluation set is a translation of a Spanish corpus, and the project's pre-registered methodology reserves deployment-grade claims for physician-labelled spans. The figures in this card are therefore development-grade measurements on a proxy. As annotated East African clinical text becomes available through the project's clinical workstream, the same evaluation harness applies unchanged, and the numbers here should be read as the estimate to be replaced by that measurement rather than as the final word.

Transfer to other languages

Nothing in the recipe is specific to these four languages beyond the training data. Extending it to another language needs three things: a synthetic clinical corpus in that language generated from the same template-and-vocabulary approach (name, place and facility lists plus the country's identifier formats), a small real-text development set for model selection and threshold fitting, and a base encoder with some pretraining coverage of the language. The last matters most. In a controlled comparison across these four languages, gains from richer training data appeared only where the backbone had pretrained on the language, and vocabulary diversification hurt where it had not. For a language none of the available encoders covers, continued pretraining on monolingual text is the first step, and the Dholuo results here are the realistic expectation for that case.

Bias, risks and limitations

Stated as measurements, on the translated-clinical test unless noted.

• Patient names are found at recall 0.50-0.73; the remainder is the principal residual risk.
• Third-party names are the weakest name category, recall 0.38-0.57. Threshold adjustment did not raise it without lowering recall elsewhere, and retraining on richer data did not move it.
• Village / area names, the largest category by span count, are found at recall 0.52-0.63. Some are mis-typed as third-party names; this is representational rather than a threshold effect.
• Names never seen in training are found at lower recall than familiar ones (table above); the gap is the memorisation effect and is the figure to plan against for real patients.
• Out-of-domain text (news) is handled at substantially lower recall (table above); thresholds fitted on clinical text do not transfer to other registers.
• Profession has 3-5 gold spans per language in the clinical test; its scores are low-powered and are reported for completeness.
• Tribe, religion and clan are evaluated only on the synthetic held-out set, which is in-distribution with training; treat those figures as upper bounds.
• Evaluation corpus. The clinical test is machine-translated from Spanish; it is a proxy for native clinical text. Physician-labelled spans in these languages did not exist at release.
• Training names. Person names in the synthetic training data are recombined from public given- and family-name vocabularies; as with any such generator, some recombinations coincide with the names of real individuals. The model emits offsets and labels only and cannot reproduce a name.

Environmental impact

Hardware: NVIDIA DGX Spark (GB10), on-premises. Usage: 7.0 GPU-hours for the full lane (sweep, ablations, finals and scoring). Emissions were not separately metered for this lane and are not estimated here.

Technical specifications

Citation

@misc{craneai2026afrimedpii,
  title  = {AfriMed-PII: Clinical de-identification models for Luganda, Kiswahili, Kinyarwanda and Dholuo},
  author = {{Crane AI Labs}},
  year   = {2026},
  note   = {Gates Foundation INV-107957, Workstream 1. Technical report forthcoming.}
}

Funding

This material is based on research funded by the Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Gates Foundation.