Nawah-Router-BERT-6M-bilingual — a router head trained on English and Arabic together
Give it a text and any categories in plain English or Arabic; it scores all of them in one forward pass. No fixed taxonomy, no separate model per language. 5,977,985 parameters.
The headline finding: English-only training did not work on this backbone
Nawah-BERT-6M-v2 was pretrained exclusively on Arabic MSA text
(fineweb-edu-ar). Training a routing head on
prompt-routing-en
alone, same code, same hyperparameters, produced a model that learned nothing: training loss
sat flat at ~1.57 for all 3 epochs, and every eval split landed at its random baseline.
Mixing in the Arabic corpus rescued it — not just held Arabic steady, but pulled English routing from pure chance to well above it:
| eval split | English-only | this model — English | this model — Arabic | Arabic-only (v2) |
|---|---|---|---|---|
| unseen_lanes | 0.210 (=random) | 0.859 | 0.929 | 0.933 |
| unseen_domain | 0.239 (=random) | 0.590 | 0.694 | 0.701 |
| unseen_axis | 0.252 (=random) | 0.547 | 0.605 | 0.600 |
| hard | 0.213 (=random) | 0.839 | 0.903 | 0.911 |
All four numbers were re-measured in the same session with eval_router_bilingual.py; the Arabic
columns reproduce the published Nawah-Router-BERT-6M-v2 card within normal run-to-run variance,
which is what makes the English-only comparison trustworthy.
Why joint training fixes it is not fully confirmed. The working hypothesis: the shared 2-layer scoring head, trained only on English gradients, got stuck in a degenerate optimum that predicts a near-uniform distribution over categories regardless of content (consistent with the flat loss staying near ln(mean lane count) ≈ 1.6 for all three epochs). The Arabic gradients, on representations the backbone was actually pretrained to produce, appear to pull the head out of that optimum early enough that it then also learns from the weaker — but not degenerate — English signal. This is a hypothesis, not a verified mechanism; it has not been tested against, say, a lower learning rate or longer warmup on English alone.
Practical takeaway
If you need English routing on a tiny Arabic-pretrained backbone, training on English data alone may not be enough — pairing it with a related-task corpus in the backbone's pretraining language was what worked here, on this backbone, at this scale. Don't assume that generalizes without testing on your own data.
Usage
from transformers import AutoTokenizer
from routing_model_bilingual import RouterModel, route # ships in this repo
M = "oddadmix/Nawah-Router-BERT-6M-bilingual"
tok = AutoTokenizer.from_pretrained(M)
model = RouterModel.from_pretrained(M)
route(model, tok, "The order is an hour late and the driver isn't answering",
["delivery enquiry", "delayed-order complaint", "payment issue"])
# lang="en"/"ar"/"auto" (default) - auto picks by script
Trained on oddadmix/prompt-routing-en (173,223 English rows) +
oddadmix/arabic-prompt-routing (207,097 Arabic rows) mixed in one training run, 3 epochs, with
train_router_head_multi.py's defaults (LR 3e-4, batch 32). Each row is wrapped in its own
language's template ("Text:/Categories:" or "النص:/الفئات:") before the shared span-pooling head
scores it — see that script's docstring.
© KAND CA 2026 — PROJECT NAWAH