50M-8192-Nawah-gemma (نواة) — Arabic Base Model, 8192-token context
A ~51.8M-parameter Gemma-3 architecture Arabic language model pretrained from scratch on 20 billion Arabic tokens with an 8192-token context window.
This is the long-context sibling of
oddadmix/50M-2048-Emhotob:
same corpus, same tokenizer, same parameter count, same token budget, same
optimizer schedule. What differs is the context window (8192 vs 2048) and
the attention layout that makes it affordable.
Base (pretrained) model: a next-token predictor with no instruction or chat tuning.
الملخص بالعربية: نموذج لغوي عربي صغير (~51.8 مليون معامل) بمعمارية Gemma-3، مُدرَّب من الصفر على 20 مليار رمز عربي بطول سياق 8192. نموذج أساس (Base) بدون ضبط تعليمات أو محادثة.
Model details
| Parameters | ~51.8M |
| Architecture | Gemma-3 (Gemma3ForCausalLM) |
| Hidden size | 512 · Layers 12 · Heads 8 (GQA, 4 KV) · head_dim 64 |
| Intermediate size | 1408 |
| Vocab size | 32000 (custom Arabic byte-level BPE, shared with the 2048 model) |
| Context length | 8192 |
| Attention | interleaved: 10 sliding (window 1024) + 2 global |
| Positional encoding | RoPE, θ = 10,000 on sliding layers, 100,000 on global layers |
| Tied embeddings | yes |
| License | Apache-2.0 |
Why interleaved attention
All-global attention at 8192 tokens costs 453M attention FLOPs per token — roughly as much as the entire rest of the forward pass, and about 4× what the 2048 model spent. Restricting five of every six layers to a 1024-token window drops that to ~13M while leaving 2 layers that see the full window, so the long-range path survives and the run costs about what the original 2048 run cost. Sliding layers use the smaller RoPE base because they only ever have to resolve 1024 positions.
Training
| Corpus | kaust-generative-ai/fineweb-edu-ar (config ar) |
| Budget | 20,000,000,000 tokens, 1 epoch over the stream |
| Packing | documents separated by </s>; documents ≥ 4096 tokens duplicated once |
| Optimizer | AdamW (fused), β = (0.9, 0.95), wd 0.1, grad clip 1.0 |
| Schedule | lr 6e-4, cosine, warmup ratio 0.02 |
| Effective batch | 32 sequences × 8192 = 262,144 tokens/step |
| Precision | bf16, torch.compile, a single consumer GPU |
The corpus is short-document heavy — median 427 tokens, and only 0.47% of
documents reach 8192 — so packing matters more than architecture for long
context here. Documents are separated by </s> (the 2048 recipe used no
separator), and documents of at least 4096 tokens are written twice to raise the
share of tokens sitting inside a genuinely long single-document span.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "oddadmix/50M-8192-Nawah-gemma"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
prompt = "اللغة العربية هي"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True,
temperature=0.8, top_p=0.9, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
Limitations
At this size and budget the model has a narrow capability range and will produce factually unreliable and sometimes incoherent text. It is not instruction-tuned or aligned, and no safety filtering has been applied. Attention is not masked at document boundaries during training, so the two global layers were exposed to cross-document context within a packed window.
© KAND CA 2026 — PROJECT NAWAH.