HAYAA Student — Arabic toxicity classifier for on-device use

A distilled, vocabulary-pruned, int8-quantized student of youssefreda9/HAYAA, built to run inside a mobile app. 651 MB -> 9.6 MB (67.8x smaller).

Labels: 0 = Safe, 1 = Toxic.

Read this before using it

The numbers below measure agreement with the teacher on synthetically generated Arabic, not accuracy against human labels. The teacher's own reported F1 is 0.924, so agreement is a ceiling on quality, not a measure of it. No human-labelled gold set was used. Treat these results as evidence that the distillation pipeline works, not that the model is fit for your traffic.

Known weaknesses:

• Franco-Arabic / arabizi (enta 3abeet) scores markedly lower than the same insult in Arabic script. Transliterated text is under-represented upstream.
• The threshold is not calibrated. The teacher trained with a 3.7x weight on the toxic class, so argmax is not a meaningful operating point. Choose a threshold on your own labelled data.
• Domain gap: upstream training data is predominantly Twitter. Comments, DMs and usernames may behave differently.
• Dialect coverage follows the distillation corpus, which was synthetic here.

Model

Results

Runtimes

Measured single-threaded, batch 1, sequence 128, on a Colab CPU. Reference for the dp and flips columns is PyTorch fp32. Expect 1.5-3x higher latency on a mid-range phone.

How it was built

1. Teacher labels raw text with full logits (soft targets) — no human labels needed
2. Vocabulary pruned to corpus tokens + all 1-2 char pieces + top 20000 by frequency
3. Student initialised from teacher: pruned embedding rows, layers None, pooler, classifier head
4. Distillation: loss = 0.8 * T^2 * KL(s/T || t/T) + 0.2 * CE, with T=3.0, 6 epochs
5. ONNX export, then dynamic int8 quantization

Method follows Hinton et al. 2015 (arXiv:1503.02531) and DistilBERT (arXiv:1910.01108).

Usage

import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("MahmoudMabrok/HAYAA-student-int8", subfolder="tokenizer")
sess = ort.InferenceSession("student_int8.onnx")

enc = tok("النص هنا", truncation=True, max_length=128,
          padding="max_length", return_tensors="np")
names = [i.name for i in sess.get_inputs()]
feed = {k: v.astype(np.int64) for k, v in enc.items() if k in names}
logits = sess.run(None, feed)[0]
e = np.exp(logits - logits.max(-1, keepdims=True))
p_toxic = (e / e.sum(-1, keepdims=True))[0, 1]

On Android: put student_int8.onnx and tokenizer/tokenizer.json in assets, and verify your Kotlin WordPiece implementation against tokenizer_fixtures.json before trusting any output. A tokenizer that is 98% right produces a classifier that demos perfectly and is broken on exactly the inputs it was built for.

Files

• student_int8.onnx — quantized, for deployment
• student.onnx — fp32
• student.tflite — LiteRT build, if conversion succeeded
• tokenizer/ — pruned tokenizer (23970 tokens)
• tokenizer_fixtures.json — frozen tokenizer behaviour for cross-language tests
• run_report.json — every metric from this run

License and attribution

Derived from youssefreda9/HAYAA, itself a fine-tune of UBC-NLP/MARBERTv2. MARBERT's authors release the checkpoints for research and direct commercial users to contact them. Fine-tuning and distillation do not reset those terms, and a permissive tag on an intermediate model cannot grant rights its author did not hold.

For commercial use, contact UBC-NLP, or rebuild from a permissively licensed teacher such as CAMeL-Lab/bert-base-arabic-camelbert-mix (Apache-2.0, covers MSA plus Egyptian, Gulf and Levantine).

@inproceedings{abdul-mageed-etal-2021-arbert,
  title = {ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic},
  author = {Abdul-Mageed, Muhammad and Elmadany, AbdelRahim and Nagoudi, El Moatez Billah},
  booktitle = {Proceedings of ACL-IJCNLP 2021},
  year = {2021},
  pages = {7088--7105}
}