Roman Urdu-English Code-Switching Language Identifier
Token classification model that identifies whether each word in a Roman Urdu/English code-switched sentence is Urdu (URD), English (ENG), or a fused hybrid word (MIX).
Model Details
- Base model: xlm-roberta-base
- Task: Token Classification (3-class: URD / ENG / MIX)
- Fine-tuned by: Zainab Binte Khalid, Code Saviours (SI-26)
- Training data: code-switching-codesaviours-si26-zainab dataset — 1,901 real, hand-collected and hand-labeled Roman Urdu/English sentences (WhatsApp messages and personally-reviewed online comments)
Training
- 8 epochs, learning rate 2e-5, warmup ratio 0.1, weight decay 0.01
- 80/20 train/test split (1,520 / 381 sentences)
- GPU (Colab T4)
Evaluation Results
| Label | F1 Score |
|---|---|
| URD | 0.967 |
| ENG | 0.947 |
| MIX | 0.000 |
Overall F1 (URD/ENG): 0.957 (95.73%)
Note on MIX: MIX applies only to single tokens that fuse Urdu and English within one word (e.g. "loadshedding", "tension-wession") — not to sentences that simply contain both languages. This pattern is genuinely rare in natural Roman Urdu/English text; the training data contains only 2 MIX-labeled tokens out of 21,368 total, which is insufficient for the model to learn this class. This reflects the rarity of the phenomenon itself rather than a data collection gap.
Intended Use
Identifying language at the word level in Roman Urdu/English code-switched text — useful for downstream NLP tasks (sentiment analysis, translation, spell-checking) that need language-aware preprocessing for this common but under-resourced language-mixing pattern.
Limitations
- Trained on informal, conversational text (WhatsApp, social comments) — may not generalize well to formal or domain-specific text (news, academic writing)
- MIX class is effectively unlearned due to extreme rarity in training data
- No standardized Roman Urdu spelling exists, so performance may vary on unusual spelling variants not seen in training