Darja-GPT-50M: Domain-Conditioned Causal LM for Algerian Darija

Darja-GPT-50M is an open-source, 54.8M parameter causal language model trained from scratch exclusively on Algerian Darija (الدارجة الجزائرية / Dziri).

Unlike previous efforts that retrofitted bidirectional masked encoders (e.g. DziriBERT) with sequence-to-sequence decoders, Darja-GPT is built natively as a modern autoregressive decoder with per-domain Low-Rank Adapters (DomainLoRALinear) and support for Classifier-Free Guidance (CFG) at inference time. This architecture solves the register-collapse and domain-leakage issues common in low-resource dialect modeling, enabling reliable steering across culinary, sports, technical, podcast, narrative, and colloquial registers.

Table of Contents

1. Key Innovations & Empirical Findings
2. Architecture Specifications
3. Domain Conditioning & Classifier-Free Guidance (CFG)
4. Custom Darija Tokenizer (Fertility Analysis)
5. Training Data & Recipe
6. Quantitative & Qualitative Evaluation
7. Quickstart & Inference Guide
8. Known Limitations & Responsible AI
9. Citation & Contact

Key Innovations & Empirical Findings

1. From FiLM to Native Domain Adapters

In earlier iterations (v3 & v4), domain steering was attempted via FiLM (Feature-wise Linear Modulation)—projecting domain embeddings to scale and shift intermediate activations. However, architectural diagnostics revealed that FiLM failed to learn distinct domain representations:

• FiLM Failure Mode: Domain weight distances remained shallow (only 15%–20% variation around the mean; overall activation RMS stalled at ~0.230), resulting in severe cross-domain contamination (e.g., cooking recipes leaking into podcast prompts).
• The v5 Adapter Breakthrough: Replacing FiLM with dedicated Domain-Specific LoRA layers (DomainLoRALinear) on attention output projections (o_proj) and MLP down-projections (down_proj) produced weight delta distances of 118%–173% of the mean. Each domain maintains independent low-rank updates ($r=16, \alpha=16$), eliminating representation compromise.

2. Classifier-Free Guidance (CFG) in an Autoregressive LM

To our knowledge, this is the first deployment of Classifier-Free Guidance for dialectal autoregressive language generation:

• During training, domain IDs are dropped to an unconditional baseline (<NULL>, $d=6$) with $p=0.10$.
• At generation time, conditional logits $\mathbf{z}{\text{cond}}$ and unconditional logits $\mathbf{z}{\text{NULL}}$ are extrapolated: $$\mathbf{z}{\text{guided}} = \mathbf{z}{\text{NULL}} + s \cdot (\mathbf{z}{\text{cond}} - \mathbf{z}{\text{NULL}})$$
• Setting guidance scale $s \in [1.5, 2.5]$ completely suppresses OOD domain leakages and sharpens domain-specific terminology without degrading grammatical flow.

3. Native Darija Morphology Tokenizer

Trained from scratch on 11,151 conversational documents, our BPE tokenizer achieves 1.474 tokens/word, delivering a 37.1% fertility reduction compared to general multilingual models like Qwen2.5 (2.342 tokens/word), drastically expanding the effective context window.

Architecture Specifications

The base model adopts modern LLaMA/Mistral-style architectural principles optimized for 50M parameter scale on a single NVIDIA T4 GPU:

Domain Conditioning & Classifier-Free Guidance (CFG)

The model features 7 distinct domain adapter slots. Each prompt can be prefixed with its domain token in text, and conditioned via the domain_ids tensor:

Custom Darija Tokenizer (Fertility Analysis)

Mainstream multilingual tokenizers (LLaMA-3, Qwen2.5, Mistral) exhibit high fertility on dialectal Arabic due to aggressive subword fragmentation of colloquial stems and code-switched terms.

We trained a dedicated BPE tokenizer on the full Algerian corpus using Hugging Face tokenizers:

• Vocabulary Size: 24,000 tokens
• Special Tokens: <pad> (0), <bos> (1), <eos> (2), <unk> (3), plus registered single-token domain tags (<cooking>, <tech>, etc.)
• Fertility Benchmark on 335 Unseen Held-out Documents:
  • Qwen2.5 Tokenizer: 2.342 tokens per word
  • Darja-GPT Custom Tokenizer: 1.474 tokens per word
  • Improvement: +37.1% efficiency gain in token density, allowing 1,024 tokens to represent ~700 words of real text rather than ~430 words.

Training Data & Recipe

1. Corpus: touati-kamel/algerian-darja-corpus (conditioned subset)

• Total Documents: 11,151 high-quality transcript documents
• Raw Volume: ~186.1 million characters / ~34.3 million words
• Token Count: ~53.3 million packed tokens
• Data Balance: Curated YouTube conversational transcripts, podcasts, cooking shows, comedy sketches, and sports commentary, cleaned and tagged with domain markers.

2. Sequence Packing & Training Setup

• Chunking Pipeline: Rather than truncating long transcripts, texts were concatenated with <eos> boundaries and packed into exact 1,024-token chunks, yielding 52,038 dense training sequences (~4.8× data yield improvement over naive truncation).
• Optimizer: AdamW ($\beta_1 = 0.9, \beta_2 = 0.95, \text{weight_decay} = 0.1$)
• Regularization: Label smoothing $\epsilon = 0.1$, gradient clipping at $1.0$
• Schedule: Cosine learning rate decay with linear warmup (peak LR $3 \times 10^{-4}$)
• CFG Dropout: 10% probability of assigning domain_ids = 6 (<NULL>) per batch
• Hardware: Single NVIDIA T4 GPU (16 GB VRAM) on Kaggle Notebooks
• Training Runtime: ~6.8 hours for 6 full epochs (4,878 optimizer steps)

Quantitative & Qualitative Evaluation

1. Held-Out Perplexity

Evaluated on 335 strictly held-out test documents (never seen during training, split with fixed seed):

• Raw Held-out Eval Loss: 5.2890
• Raw Held-out Perplexity: 198.15 (For context, training-time best eval loss was ~5.978; the packed held-out perplexity demonstrates zero overfitting and solid generalizability across dialects).

2. Weight Diagnostic (FiLM vs. LoRA Adapters)

Direct inspection of weight-delta Frobenius norms $|\Delta W|_F$ across all 12 layers:

3. Qualitative Steering Regimes

Each domain was evaluated across 5 controlled experimental regimes:

1. notag_null: Prompt with no text tag, evaluated on unconditional base (<NULL>).
2. tag_null: Prompt with text tag, evaluated on unconditional base.
3. tag_correct: Prompt with text tag + matching domain adapter.
4. tag_mismatch: Causal test — Prompt text tag matches Domain A, but adapter is set to Domain B.
5. tag_correct+CFG: Prompt with text tag + matching adapter guided via CFG ($s=2.5$).

Key Findings from Empirical Samples:

• Causal Override Proved (Mismatch Test): Feeding <cooking> prompt text into the <tech> adapter immediately injected electronics pricing, battery life, and Bluetooth mentions into cooking sentences, proving the adapter causally controls generation semantics rather than relying purely on text tag autoregression.
• The "Mahrez" Sports Evaluation: In 5/5 independent seeded sports CFG draws, Riyad Mahrez appeared as the central subject. Detailed analysis confirmed this reflects true corpus distribution (Mahrez accounts for >80% of modern Algerian sports media mentions) rather than degenerate looping: each sample paired Mahrez with completely unique sets of international clubs (Barcelona, Man City, Chelsea) and national teams (Burkina Faso, Austria, Senegal, Egypt).
• Narrative Coherence (<story>): Produced sustained multi-sentence folklore and personal narratives in 10/10 samples with no phrase-level degeneracy.

Quickstart & Inference Guide

Because Darja-GPT-50M-v5 uses custom rotary and domain-adapter layers, generation requires registering the configuration or using the standalone model file.

Installation

pip install torch transformers datasets sentencepiece

Complete Generation Script with Classifier-Free Guidance (CFG)

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

# 1. Load Tokenizer & Model
MODEL_ID = "touati-kamel/darja-gpt-50m"
TOKENIZER_ID = "touati-kamel/darja-tokenizer-50m"

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, trust_remote_code=True).to(device)
model.eval()

# Domain Mapping
DOMAIN_TO_IDX = {
    "<general>": 0,
    "<cooking>": 1,
    "<tech>": 2,
    "<sports>": 3,
    "<podcast>": 4,
    "<story>": 5,
    "<NULL>": 6
}
NULL_DOMAIN_IDX = 6

@torch.no_grad()
def generate_darja_cfg(
    prompt: str,
    domain_tag: str = "<cooking>",
    guidance_scale: float = 2.0,
    max_new_tokens: int = 120,
    temperature: float = 0.8,
    top_p: float = 0.9,
    repetition_penalty: float = 1.25
):
    full_prompt = f"{domain_tag} {prompt}"
    input_ids = tokenizer.encode(full_prompt, return_tensors="pt").to(device)
    
    cond_domain = torch.tensor([DOMAIN_TO_IDX[domain_tag]], dtype=torch.long, device=device)
    uncond_domain = torch.tensor([NULL_DOMAIN_IDX], dtype=torch.long, device=device)
    generated = input_ids

    for _ in range(max_new_tokens):
        if generated.size(1) >= model.config.max_position_embeddings:
            break
            
        # Parallel forward passes for CFG
        logits_cond = model(input_ids=generated, domain_ids=cond_domain).logits[:, -1, :]
        logits_uncond = model(input_ids=generated, domain_ids=uncond_domain).logits[:, -1, :]
        
        # Classifier-Free Guidance extrapolation
        logits = logits_uncond + guidance_scale * (logits_cond - logits_uncond)
        
        # Repetition penalty
        for token_id in set(generated[0].tolist()):
            if logits[0, token_id] > 0:
                logits[0, token_id] /= repetition_penalty
            else:
                logits[0, token_id] *= repetition_penalty
                
        # Temperature & Top-p (Nucleus) sampling
        logits = logits / max(temperature, 1e-5)
        probs = F.softmax(logits, dim=-1)
        sorted_probs, sorted_indices = torch.sort(probs, descending=True)
        cumulative_probs = torch.cumsum(sorted_probs, dim=-1)
        
        cutoff = cumulative_probs > top_p
        cutoff[..., 1:] = cutoff[..., :-1].clone()
        cutoff[..., 0] = False
        sorted_probs[cutoff] = 0.0
        sorted_probs = sorted_probs / sorted_probs.sum(dim=-1, keepdim=True)
        
        next_token = sorted_indices.gather(-1, torch.multinomial(sorted_probs, num_samples=1))
        generated = torch.cat([generated, next_token], dim=1)
        
        if next_token.item() == tokenizer.eos_token_id:
            break

    return tokenizer.decode(generated[0], skip_special_tokens=True)

# Example 1: Culinary Recipe in Algerian Darija
cooking_output = generate_darja_cfg(
    prompt="اليوم غادي نديرو وصفة بسيطة وسهلة، خاصنا",
    domain_tag="<cooking>",
    guidance_scale=2.2
)
print("--- Cooking Output ---")
print(cooking_output)

# Example 2: Sports Commentary
sports_output = generate_darja_cfg(
    prompt="البارح كانت مباراة كبيرة وقوية بزاف",
    domain_tag="<sports>",
    guidance_scale=2.5
)
print("\n--- Sports Output ---")
print(sports_output)

Known Limitations & Responsible AI

1. Parameter Scale (54.8M):
   • This model is intended for fast stylistic generation, grammatical exploration, dialectal modeling, and low-compute deployment.
   • It is not an instruction-tuned assistant (not SFT aligned) and cannot perform complex multi-step reasoning, mathematical calculations, or factual QA.
2. Entity Bias:
   • The sports domain exhibits high name-recurrence (e.g. Riyad Mahrez, Youcef Belaïli) reflecting YouTube sports talk media frequency in Algeria.
3. Orthographic Heterogeneity:
   • Algerian Darija is traditionally an unstandardized spoken dialect with mixed Arabic and Latin (Arabizi) orthographies. While the tokenizer accommodates both, Arabic script accounts for ~90% of the training corpus.
4. Code-Switching:
   • In technical and business podcast domains, the model naturally code-switches into French and English loanwords, reflecting authentic Algerian speech patterns.

Citation & Contact

If you use Darja-GPT-50M or the Algerian Darja Corpus in your research or project, please cite:

@misc{touati2026darjagpt50m,
  author       = {Kamel Touati},
  title        = {Darja-GPT-50M: Domain-Conditioned Causal Language Model for Algerian Darija with Classifier-Free Guidance},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/touati-kamel/darja-gpt-50m}}
}

@misc{touati2026darjacorpus,
  author       = {Kamel Touati},
  title        = {Algerian Darja Corpus: Multi-Domain Conversational Transcripts in Algerian Arabic},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/touati-kamel/algerian-darja-corpus}}
}

Developer: Kamel Touati (Hugging Face Profile)
License: Apache 2.0