Multilingual Toxicity Classifier (XLM-RoBERTa-base V5)
Binary toxicity classifier for Turkish, Arabic, and English text, built on XLM-RoBERTa-base. V5 focuses on real-world chat robustness: Arabic dialects, Arabizi / Franco-Arabic, obfuscation (diacritics, tatweel, elongation, homoglyphs), implicit toxicity, and reduced false positives on dialectal greetings and negation / counter-speech.
Author
- Görkem Yıldız · GitHub: gorkem371
Model Details
| Property | Value |
|---|---|
| Base Model | xlm-roberta-base (280M params) |
| Task | Binary text classification (not-toxic / toxic) |
| Languages | Turkish, Arabic (MSA + dialects), English |
| Labels | 0 = not-toxic, 1 = toxic |
| Training | Focal loss (γ=2) + inverse class weights, bf16, cosine schedule, text normalization |
Performance
| Metric | Score |
|---|---|
| Validation F1 | 0.907 |
| Validation Accuracy | 0.906 |
On an internal hard adversarial "frontier" set (dialects, Arabizi, obfuscation, implicit, negation), V5 scores 88% vs 38% for V4 and 41% for V3 — while preserving V4's explicit-toxicity and Arabizi gains.
⚠️ Important: apply the same text normalization at inference
V5 was trained on normalized text (strip Arabic diacritics/tatweel, unify alef/ya, collapse
character elongation). You must apply the same normalization before tokenizing, or robustness
to diacritics/obfuscation degrades. The text_normalize.py used for training is included in this repo.
import re, unicodedata, torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
_HARAKAT = re.compile("[ؐ-ًؚ-ٰٟۖ-ۜ۟-۪ۨ-ۭـ]") # diacritics + tatweel
_ALEF = re.compile("[آأإٱ]")
def normalize_text(t):
t = unicodedata.normalize("NFKC", t)
t = _HARAKAT.sub("", t)
t = _ALEF.sub("ا", t)
t = t.replace("ى", "ي").replace("ؤ", "و").replace("ئ", "ي")
t = re.sub(r"(.)\1{2,}", r"\1\1", t) # collapse 3+ repeats
return re.sub(r"\s+", " ", t).strip()
name = "gorkem371/toxicity-classifier-xlmr-base-v5"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()
def classify(text):
text = normalize_text(text)
inputs = tok(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
probs = torch.softmax(model(**inputs).logits, dim=-1)[0]
label = model.config.id2label[int(probs.argmax())]
return label, float(probs.max())
print(classify("يا كلب اخرس")) # ('toxic', ...)
print(classify("this is fucking awesome")) # ('not-toxic', ...) (profanity-as-emphasis)
print(classify("lan naber nasılsın")) # ('not-toxic', ...)
What V5 handles well
- Arabic dialects: MSA, Egyptian, Levantine, Gulf, Iraqi, Maghrebi, Sudanese, Libyan (toxic + clean greetings)
- Arabizi / Franco-Arabic (Latin+numbers) in both directions
- Obfuscation: diacritics (tashkeel), tatweel, character elongation, homoglyphs, dotted/spaced letters, emoji
- Negation / counter-speech and profanity-as-positive-emphasis (fewer false positives)
- Self-deprecation and casual interjections (e.g. Turkish "lan")
Known limitations
- Veiled / polite threats with no explicit words ("I know where you live")
- Sarcasm / irony where the surface is positive
- Coded bias / dog-whistles
- Some benign homographs (e.g. talking about a pet dog/pig can occasionally false-positive)
- Turkish negation ("X is not stupid") can still false-positive
- In-group / reclaimed slurs are flagged (a policy gray area)
Training approach
Base data (~100K balanced samples across TR/AR/EN from public datasets) plus iteratively hand-crafted targeted sets that close specific failure modes found by adversarial testing (Arabizi, "يا [insult]" vocative, implicit toxicity, dialect greetings, negation, homographs). Targeted samples are repeated 10× and merged onto the base; all text is normalized at train and inference time.
License
Apache 2.0