RuqLM — small Arabic language models

▶ Try it live: ruqlm.web.app — runs entirely in your browser, no server, no signup needed for guest access.

A ladder of five Arabic language models, 0.74M to 29.89M parameters, trained from scratch on a corpus of 50,003 Arabic stories under a fixed budget of 76.8M training tokens. The tokenizer, the corpus and the weights were all built for this project.

Weights are full-precision ONNX — no quantisation. Perplexity matches the original PyTorch checkpoints to three decimals (9.2009 vs 9.2009), so these are the trained models, not an approximation of them.

The ladder

Decoder-only transformer: RMSNorm, RoPE, SwiGLU, tied embeddings, 512-token context. Tokenizer is a byte-level BPE with an 8,192 vocabulary trained on the same corpus (1.411 tokens per word).

Usage

pip install onnxruntime tokenizers huggingface_hub numpy

Colab has none of these preinstalled except numpy, so run the line above first (prefix it with ! in a notebook cell).

import re, unicodedata
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

REPO, EOS = "Ruqiya/ruqlm", 2

tok  = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json"))
sess = ort.InferenceSession(hf_hub_download(REPO, "ruq-30m.onnx"))

# The tokenizer was trained on normalised text: diacritics and tatweel
# removed. Skipping this step gives a different, worse tokenisation.
_DIACRITICS = re.compile(r"[\u064b-\u0652\u0670\u0653-\u0655]")
_ZERO_WIDTH = re.compile(r"[\u200b-\u200f\u202a-\u202e\ufeff]")

def normalize(text):
    text = unicodedata.normalize("NFC", text)
    text = _ZERO_WIDTH.sub("", text).replace("ـ", "")
    return re.sub(r"\s+", " ", _DIACRITICS.sub("", text)).strip()

def generate(prompt, max_new_tokens=120, temperature=0.85, top_k=50, seed=None):
    """The model outputs logits for the last position only; sampling is here."""
    rng = np.random.default_rng(seed)
    ids = tok.encode(normalize(prompt)).ids
    if ids and ids[-1] == EOS:          # drop the trailing </s> so it continues
        ids = ids[:-1]

    for _ in range(max_new_tokens):
        logits = sess.run(None, {"input_ids": np.array([ids], dtype=np.int64)})[0][0]
        logits = logits.astype(np.float64) / temperature

        kth = np.partition(logits, -top_k)[-top_k]      # top-k filter
        logits[logits < kth] = -np.inf

        probs = np.exp(logits - logits.max())
        probs /= probs.sum()

        nxt = int(rng.choice(len(probs), p=probs))
        if nxt == EOS:
            break
        ids.append(nxt)

    return tok.decode(ids)

print(generate("كان يا ما كان"))

Output (ruq-5m, seed 0):

كان يا ما كان في حديقة بيت. كان النجار يعمل بجد ليطعم عائلته. وفي يوم من الأيام، وأثناء عمله، وجد النجار سلحفاة صغيرة ضائعة…

PyTorch

The same weights are published as safetensors alongside the architecture module, for continued training or fine-tuning:

import json, re, unicodedata, torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

REPO, EOS = "Ruqiya/ruqlm", 2

tok = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json"))

_DIACRITICS = re.compile(r"[\u064b-\u0652\u0670\u0653-\u0655]")
_ZERO_WIDTH = re.compile(r"[\u200b-\u200f\u202a-\u202e\ufeff]")

def normalize(text):
    text = unicodedata.normalize("NFC", text)
    text = _ZERO_WIDTH.sub("", text).replace("\u0640", "")     # tatweel
    return re.sub(r"\s+", " ", _DIACRITICS.sub("", text)).strip()

# the architecture module has to be on the path before it can be imported
hf_hub_download(REPO, "modeling_ruqlm.py", local_dir=".")
from modeling_ruqlm import RuqLM, ModelArgs

cfg = json.load(open(hf_hub_download(REPO, "configs.json")))["ruq-5m"]
model = RuqLM(ModelArgs(**cfg))

state = load_file(hf_hub_download(REPO, "ruq-5m.safetensors"))
state["lm_head.weight"] = state["tok_emb.weight"]   # embeddings are tied
model.load_state_dict(state)
model.eval()

ids = tok.encode(normalize("كان يا ما كان")).ids[:-1]   # drop the trailing </s>
out = model.generate(torch.tensor([ids]), max_new_tokens=120,
                     temperature=0.85, top_k=50, eos_id=EOS)
print(tok.decode(out[0].tolist()))

lm_head and tok_emb are the same tensor, so only one is stored and the tie is restored on load. modeling_ruqlm.py depends on nothing but torch.

Interface

The graph takes input_ids of shape [1, seq] and returns logits for the last position only, shape [1, 8192]. The sampling loop lives outside the model, so temperature and top-k can change without re-exporting.

In the browser

<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.min.js"></script>
<script type="module">
import { AutoTokenizer } from
  'https://cdn.jsdelivr.net/npm/@huggingface/transformers/+esm';

const REPO = 'https://huggingface.co/Ruqiya/ruqlm/resolve/main';
const tok  = await AutoTokenizer.from_pretrained('Ruqiya/ruqlm');
const sess = await ort.InferenceSession.create(`${REPO}/ruq-5m.onnx`);

let ids = Array.from((await tok('كان يا ما كان')).input_ids.data, Number);
if (ids.at(-1) === 2) ids = ids.slice(0, -1);           // drop </s>

for (let i = 0; i < 100; i++) {
  const input = new ort.Tensor('int64', BigInt64Array.from(ids, BigInt),
                               [1, ids.length]);
  const { logits } = await sess.run({ input_ids: input });
  // greedy for brevity; sample with temperature and top-k for better text
  const next = logits.data.indexOf(Math.max(...logits.data));
  if (next === 2) break;
  ids.push(next);
}
console.log(tok.decode(ids, { skip_special_tokens: true }));
</script>

The tokenizer normalises text the same way the Python example does, so no separate normalisation step is needed here.

Fine-tuning

The safetensors carry weights only — no optimiser state — so a fine-tune starts from a fresh optimiser:

model.train()
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)

# input_ids: LongTensor [batch, seq]; the model shifts internally, so labels
# are the inputs themselves
logits, loss = model(input_ids, labels=input_ids)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step(); opt.zero_grad()

Context length is 512. Chat-role tokens <|user|> (id 4) and <|assistant|> (id 5) are already reserved in the vocabulary, so instruction tuning needs no tokenizer change.

Limitations

These are base text-completion models, not chat models. They were trained on Arabic stories only, so they continue what you start; they do not answer questions or follow instructions, having had no instruction tuning (SFT) or RLHF. Expect narrative, not answers.

Morphology and orthography are strong. Semantic and referential coherence are weaker — which is what the accompanying study measures, and what these models were built to quantify rather than to solve. Even the largest rung abandons 5.9% of the characters it introduces, where the training corpus abandons 0.0%.

The corpus is synthetic, generated by two teacher models: ALLaM 2.7B (75%) and Qwen3.6-27B (25%), with quality gates rejecting foreign characters and template openings. Both teachers are Apache-2.0.

What the models are for

They were built to answer a measurement question: across a 40× range of parameters at fixed data, which linguistic competence scales and which saturates?

Capacity buys coherence, not vocabulary.

Paper

In preparation. The paper covers the corpus construction, the evaluation framework, and every measurement reported here.

@misc{binsafi2026ruqlm,
  title  = {What Scales in a Small Arabic Language Model?
            Morphology Saturates, Coherence Does Not},
  author = {Bin Safi, Ruqiya},
  year   = {2026},
  note   = {Preprint}
}