algerianDeBERTa

algerianDeBERTa is a ~60M-parameter masked language model built specifically for the Algerian linguistic space. It was trained on a custom Algerian corpus with a purpose-built tokenizer, and is designed to be a small, fast, fine-tunable backbone for downstream Algerian NLP tasks (sentiment, intent, classification, retrieval, NER).

Headline: it matches ~96–98% of DziriBERT's macro-F1 on Algerian sentiment benchmarks while using ≈ half the parameters and half the memory (231 MB vs 475 MB).

Why another Algerian model?

Algerian is a genuinely hard, low-resource setting:

• Three+ languages in one sentence — Arabic, French and Tamazight vocabulary mixed freely.
• Two scripts — the same word is written in Arabic letters and in Latin "Arabizi" (e.g. wach rak / واش راك), often with digits standing in for sounds (3 = ع, 7 = ح, 9 = ق).
• No standard orthography — spelling is phonetic and varies per writer and per region.

General multilingual models and even MSA-centric Arabic models underperform here. algerianDeBERTa tackles this with (1) a tokenizer learned directly on Algerian text and (2) DeBERTa-v2's disentangled attention, which is strong at modeling the messy, non-canonical token order of dialectal writing — all in a deliberately small footprint.

Highlights

Performance vs DziriBERT

DziriBERT is the reference Transformer for Algerian dialect and the prior state of the art on Algerian text classification (it beats mBERT, AraBERT, CAMeLBERT, QARiB and MARBERT on these tasks). It is the right yardstick.

Both models were fine-tuned identically (same heads, same hyper-parameters , LR 2e-5, max length 128, seed 42, 85/15 split) on three Algerian sentiment datasets, and compared on macro-F1:

DziriBERT keeps a small edge in raw macro-F1 (≈ 2–3 points), which is expected from a model with roughly twice the parameters. The point of algerianDeBERTa is the trade-off: near-parity quality at half the size.

Memory footprint

That ~51% reduction in size means lower memory, faster inference, and cheaper fine-tuning — useful for edge deployment and for stacking the model into multi-stage pipelines.

How to use

Fill-mask

from transformers import pipeline

fill = pipeline("fill-mask", model="81melody/algerianDeBERTa")
fill("راني [MASK] بزاف اليوم")
fill("salam, wach rak [MASK]?")

Get embeddings / features

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("81melody/algerianDeBERTa")
model = AutoModel.from_pretrained("81melody/algerianDeBERTa")

text = "نحب نشري دار في وهران"
inputs = tok(text, return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)
# CLS / pooled representation
cls = out.last_hidden_state[:, 0]
print(cls.shape)  # (1, 512)

Fine-tune for classification

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("81melody/algerianDeBERTa")
model = AutoModelForSequenceClassification.from_pretrained(
    "81melody/algerianDeBERTa", num_labels=3
)
# then train with transformers.Trainer on your Algerian dataset

Model details

Training

Data

Pre-trained from scratch on a custom Algerian corpus (~115 MB of cleaned text) assembled from public Algerian web and social-media content, including ~45k YouTube comments. The corpus deliberately spans the full Algerian register:

• Algerian Darja in Arabic script
• Arabizi (Latin-script Algerian with digit substitutions)
• French and French↔Arabic code-switching
• some Modern Standard Arabic

Text was cleaned and normalized (deduplication, noise/boilerplate removal) before training. The tokenizer was trained on this same corpus so that frequent Darja/Arabizi sub-words get dedicated tokens instead of being shattered into bytes.

Procedure

The held-out masked-LM loss decreases steadily over training, from ~3.46 to 3.40.

Models built on algerianDeBERTa

Intended uses & limitations

Intended uses

• A base encoder to fine-tune on Algerian-dialect downstream tasks: sentiment, intent, topic/spam classification, NER, retrieval/reranking.
• Masked-token infilling and feature extraction for Algerian text.

Limitations & biases

• It is a base MLM, not an instruction-tuned or chat model — it does not "answer" prompts; it needs fine-tuning for most tasks.
• Trained on public social-media text, so it can reflect the biases, slang, and toxicity present in that data.
• Coverage is strongest for the Darja/Arabizi/French mix it was trained on; performance on pure MSA or on Tamazight may be weaker than dedicated models.
• The corpus is region-skewed toward the dialects most common online; some regional variants are under-represented.

Citation

@misc{himeur2026algerianDeBERTa,
  title        = {algerianDeBERTa: A Compact DeBERTa-v2 Encoder Pre-trained from Scratch
                  for Algerian Darja, Arabizi and Code-Switching},
  author       = {Himeur, Ayoub},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/81melody/algerianDeBERTa}
}

Acknowledgements

Benchmarked against DziriBERT (Abdaoui et al.), the prior state of the art for Algerian dialect modeling.

License

Released under the Apache 2.0 license.