MeridianPII — Telugu PII Detection (v1)
On-device PII detection for Telugu (తెలుగు script), romanized/code-switched Telugu, and Indian-English text. Fine-tuned from microsoft/Multilingual-MiniLM-L12-H384, vocabulary-trimmed to a 54 MB INT8 ONNX that runs in the browser via transformers.js. A Telugu locale pack in the MeridianPII family — route Telugu text here; see meridianpii-hi-v2 for Hindi.
Extends Rampart's architecture to Telugu, a script Rampart's tokenizer cannot represent at all.
Evaluation
Span-level, on a frozen 1,500-row Telugu test set (Telugu-script · romanized · Indian-English), template-disjoint from training. Recall is the headline — a missed entity is a leak.
| Model | Telugu-script recall | Overall recall | Size · runs |
|---|---|---|---|
| MeridianPII te-v1 | 97.8% | 98.7% | 54 MB · on-device |
| Piiranha | 61.2% | 67.8% | 303 MB · server |
| Rampart | 51.9% | 69.0% | 15 MB · on-device |
| GLiNER-PII | 45.1% | 65.4% | 300 MB · server |
| Presidio | 38.0% | 53.1% | ~600 MB · server |
| mBERT NER | 32.1% | 35.8% | 129 MB · server |
No general-purpose tool clears 62% on Telugu script; the purpose-built pack is +37 points over the best of them, at a fraction of the size and fully on-device.
Recommended pipeline
Ship alongside the deterministic recognizer layer (regex + checksums) that owns structured IDs — Aadhaar (Verhoeff), PAN, IFSC, GSTIN (mod-36), vehicle registration, voter ID, +91 phones, email, URL (national formats, shared across Indian locales). These are premasked in training so the model spends capacity on Telugu names/addresses/cities/contextual PII. Reference: github.com/plingampally/meridianpii. Keep-set CITY/STATE/ZIP are detected but retained.
Usage
import { pipeline } from '@huggingface/transformers';
const pipe = await pipeline('token-classification', 'plingampally/meridianpii-te-v1', { dtype: 'q8' });
const out = await pipe('నా పేరు వెంకట రావు, ఫోన్ 9876543210', { aggregation_strategy: 'simple' });
Use aggregation_strategy="simple". ONNX: onnx/model_quantized.onnx (INT8). Recommended confidence floor 0.15 (recall-biased).
Limitations
- Scope is Telugu-locale by design (locale pack); route other languages to their packs.
- CITY/STATE frequently interchanged (both keep-set → no redaction impact).
- Non-Indian geography out of scope. Structured IDs need the deterministic layer.
- Test set is synthetic; evaluate on your own data.
Labels (35 BIO = O + B-/I- × 17)
GIVEN_NAME, SURNAME, EMAIL, PHONE, URL, TAX_ID, BANK_ACCOUNT, ROUTING_NUMBER, GOVERNMENT_ID, PASSPORT, DRIVERS_LICENSE, BUILDING_NUMBER, STREET_NAME, SECONDARY_ADDRESS, CITY, STATE, ZIP_CODE (EMAIL/URL/TAX_ID/ROUTING premasked at serve time).
Training & license
Base microsoft/Multilingual-MiniLM-L12-H384 (Apache 2.0), vocab trimmed 250k→89k pieces (Latin + Telugu + digits, 100% output parity). Data: ~25k rows — Telugu-script / romanized-Telugu / Indian-English synthetic (template expansion over Telugu name/geo/format fillers). NFC normalization (never NFKD — it strips Telugu gunintams/matras). CC BY 4.0. Derives from Rampart (CC BY 4.0) — architecture, schema, premasking methodology.