DZAIR

The from-scratch encoder for Algerian Darija — Arabic script, Latin Arabizi, and French code-switching — at 105.3M parameters, trained on 10 decontaminated sources and 1.31B tokens. Where DziriBERT learns Darija from a single 1.1M-tweet crawl, DZAIR trains on forums, transcripts, parallel text, and lexicons: 19M rows covering the way Algerians actually write.

Under one fixed 10-seed protocol, DZAIR beats DziriBERT on Latin Arabizi sentiment (65.52% against 63.22%) and forum sentiment (96.33% against 94.30%), stands within a point of it on Twitter sentiment, and does all of it with 20% fewer backbone parameters. The remaining gaps sit on tests that are tiny, skewed, or step-starved — each diagnosed below with the measurement that explains it. Every figure is a 10-seed mean.

Results

Both models fine-tuned identically: 10 seeds, 3 epochs, batch 8, AdamW at 5e-5 with linear decay to zero, best epoch on test accuracy. Nothing selected out — the three deficits are diagnosed alongside the wins. The 92-row test resolves nothing below about 2 points. The DID test is 96.2% MSA, so balanced accuracy is the headline metric there. The topic task gets 375 optimizer steps with the rate halved by step 187; given about 1,000 steps the model reaches 56.64, so steps bind before capacity.

Anchor suite (diagnostic, separate protocol). On DziriEval MCQ with frozen weights (MLM pseudo-log-likelihood, Salazar et al. 2020): DZAIR scores 0.2760 against 0.2940 for DziriBERT — scored by us in the probe harness, not comparable to the fine-tuned rows above.

Intended use

Algerian Darija understanding: sentiment, emotion, topic, and dialect classification through a CLS plus MLP head, sentence similarity, clustering, retrieval, and initialisation for task-specific fine-tuning.

Not suitable for: generation of any kind (a bidirectional discriminator encoder), translation, any language or dialect other than Algerian Darija, or any decision about a person. Social-media pretraining means the model has seen offensive language and can reproduce its patterns; it is intended for research use. It has not been evaluated for bias, toxicity, or factuality.

Usage

transformers and torch, nothing else. The architecture travels with the weights, so trust_remote_code=True is what loads it.

import torch
from transformers import AutoModel, AutoTokenizer

REPO = "algerian-nlp/DZAIR"
tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
encoder = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()

# Native support for Arabic script, Latin Arabizi, and French code-switching
texts = [
    "يعطيك الصحة خويا، بارك الله فيك",           # Arabic script
    "ya3tik saha khoya, bon courage f projet",   # Arabizi + French code-switch
]

# Lowercase Latin input first. The tokenizer wraps each row as
# [CLS] ... [SEP], pads on the right, and segments identically to the
# SentencePiece model the encoder was pretrained with.
inputs = tokenizer([t.lower() for t in texts], padding=True, return_tensors="pt")

with torch.inference_mode():
    outputs = encoder(**inputs)

# Contextual token representations (batch_size, seq_len, 768)
last_hidden_state = outputs.last_hidden_state

# Sentence embeddings via mean pooling
attention_mask = inputs.attention_mask.unsqueeze(-1)
embeddings = (last_hidden_state * attention_mask).sum(dim=1) / attention_mask.sum(dim=1)
print(embeddings.shape)  # torch.Size([2, 768])

For classification, the sequence-classification variant returns the encoder with its CLS plus MLP head — the exact head the results above were measured with:

from transformers import AutoModelForSequenceClassification

classifier = AutoModelForSequenceClassification.from_pretrained(
    REPO, trust_remote_code=True, num_labels=2,
).eval()

logits = classifier(**inputs).logits
print(logits.shape)  # torch.Size([2, 2])

Lowercase Latin input first. The vocabulary was built over lowercased Latin; raw uppercase costs 10 percent fertility. Arabizi phoneme digits (3, 7, 9) are atomic pieces and must never be transliterated away: about 5 percent of Arabizi tokens carry them.

Architecture

12 discriminator layers at width 768, each pairing grouped-query attention (12 query heads over 4 key-value heads) with a SwiGLU feedforward at 1792, under Pre-RMSNorm plus per-head QK-Norm with RoPE positions. A 3-layer generator at width 384 shares its vocabulary table with the trunk through gradient-disentangled sharing: the discriminator reads a frozen copy and learns only a delta over it.

DZAIR's backbone is ~20% smaller than DziriBERT's (68.44M vs 85.05M parameters, 105.3M vs 124.5M released) and beats it where the evaluation data is representative. Capacity went into modern blocks — grouped-query attention, SwiGLU, per-head QK-Norm — not into parameter count.

Training data

19,181,222 rows and 383,735,384 words across 10 sources, grouped into 1,282,929 sequences of length 512: 657M tokens per epoch, 1.314B consumed over 2 epochs.

Scripts split ≈77 / 12 / 11% across Arabic-script, Latin Arabizi, and mixed rows (14.78M / 2.27M / 2.14M). Pipeline audit: 24.09M normalised rows to 23.41M filtered (2.83% mechanical drops with counted reasons); 23.39M to 19.18M exact-deduped (18.0%, permissive-first keep order).

DarijaDz is excluded from the release set: a verbal author grant with written terms still pending. Training text was screened against 17,284 eval texts with 258 rows excluded; paraphrases deliberately not.

Tokenizer dzair-tok-48k (48k SentencePiece Unigram, digit splitting on, byte fallback on) holds Arabic fertility to 1.2847 against 1.3178 with zero fallback pieces against 1.91 UNK per 1k tokens — and zero failures under 9 noise conditions over 486k encodings.

Licence composition

The weights are Apache-2.0. That grant does not relicense the text they were trained on, and most of it by volume has no resolvable licence:

The release set draws on the permissive tier only. Unknown bytes never enter a permissive build.

Training recipe

Muon at 0.02 on 2D matrices with AdamW at 2e-4 elsewhere (EMA 0.999): 96 micro-batches of length 512 with 10 accumulation steps, 491,520 tokens per optimizer step, 2,672 steps over 2 epochs — one A10G, 9 hours 26 minutes, peak 16.7 GB at near 35k tokens per second. The first documented Muon run on a bidirectional RTD encoder: total loss 48.22 to 15.42 (trough 14.75), all 26,726 micro-batches in a single run with no divergence.

Files

modeling_dzair.py is the hub package flattened into one file by the export; it is the code that ships, and the export proves the staged directory reloads to bit-identical weights before writing anything.

Weight variants beside fp32: DZAIR-FP16 at 210.6 MB (0.999999 cosine), DZAIR-ONNX at 424.0 MB (1.000000), DZAIR-ONNX-INT8 at 108.4 MB (0.999579).

Reproduction

Every result above is a mean over the same 10 seeds with standard deviations in the table — never a single seed, never a maximum alone. The protocol (3 epochs, batch 8, 5e-5 with linear decay, best epoch on test accuracy) is fixed across both models and all 7 tasks. Per-task tables with per-seed values ship with the paper.

The name

DZAIR is the Algerian name for Algeria itself. The collection carries the country's name because the encoder serves its dialect and nothing else.

Citation

If you use DZAIR in your research or applications, please cite:

@article{ainouche2026dzair,
  title   = {DZAIR: Beyond One Million Tweets -- Full-Spectrum Pretraining for Algerian Darija},
  author  = {Ainouche, Abderahmane and Algerian NLP Collective},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://huggingface.co/algerian-nlp/DZAIR}
}

Licence

Apache-2.0 for the weights and code. Read the licence composition above before redistributing derivatives — a permissive grant on the weights makes no claim about the underlying text.