Model for Code-Switched Roman Urdu & English
This model is a fine-tuned version of XLM-RoBERTa (base) trained for token-level language identification in Roman Urdu and English text.
Intended Use & Task
- Task: Token Classification (Language Identification)
- Labels:
0 / URD: Roman Urdu token1 / ENG: English token2 / MIX: Code-switched / Hybrid / Localized token
Performance & Evaluation Metrics
Evaluated on test set tokens:
| Class | Precision | Recall | F1-Score |
|---|---|---|---|
| URD | 1.00 | 0.98 | 0.99 |
| ENG | 0.96 | 1.00 | 0.98 |
| MIX | 0.89 | 0.80 | 0.84 |
| Accuracy | — | — | 0.98 |
How to Use
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
model_id = "Sumair-Parveiz/code-switching-codesaviours-si26-sumair"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)
nlp = pipeline("token-classification", model=model, tokenizer=tokenizer)
example = "Main abhi sochoing mode mein hoon"
print(nlp(example))