Ravaan-AR-70M — an autoregressive Urdu baseline, released as a control

70M parameters, trained from scratch on a purpose-built 85.4M-token Urdu corpus. On its own terms it is a competent small Urdu language model. Its reason for existing is to be the matched baseline for ravaan-diff-70m — identical corpus, identical model, identical task mixture and tokenizer, so that the two can be compared without confounds.

If you want the better Urdu model, take the diffusion one. If you want to check that comparison yourself, you need this one.

The comparison

Held-out validation bits-per-byte on native Urdu, lower is better:

The released checkpoint is deliberately not the last one

This is the 4-epoch checkpoint, not the 16-epoch one the run finished at. Its full curve:

Autoregressive training on this corpus turns at 4 epochs. Past that, more compute makes the model monotonically worse on held-out text — 0.7774 → 0.7887 → 0.8690 — while its training loss keeps falling. That is memorization, and it is the central finding the paired release exists to show: the diffusion twin, given the same corpus, descends for 64 epochs and never turns.

Taken further this gets dramatic. On this project's other, smaller corpus an AR model run for 426 epochs reached 3.6143 bpb — worse than knowing nothing, 9.98 bits per token worse than a uniform distribution over the vocabulary, while reciting 24.9% of its 32-token windows verbatim from its training data. A confidently-wrong memorizer can be arbitrarily worse than chance. That checkpoint is not released; this one is, because 4 epochs is where this arm was actually good.

Usage

Not a transformers architecture; AutoModel will not load it. Inference code ships here.

pip install torch sentencepiece safetensors
python generate.py --prefix "پاکستان کی معیشت" --tokens 160

from ravaan_infer.loader import load
from ravaan_infer.sampling import SamplingConfig, build_generator, generate, prompts
import sentencepiece as spm

arm = load(".", device="cpu")
sp = spm.SentencePieceProcessor(model_file=str(arm.tokenizer_path))

prefix = sp.encode("پاکستان کی معیشت")
prompt = prompts.lm(arm.framing, "ar", prefix=prefix)
out = generate(
    arm.model, prompt,
    config=SamplingConfig(temperature=1.0, top_p=0.95),
    max_new_tokens=160, forbid=arm.forbidden,
    eos_id=arm.eos_id, generator=build_generator("cpu", 0),
)
print(sp.decode([int(t) for t in out.tokens[0] if t >= 16]))

Unlike its diffusion twin this arm chooses its own output length and stops at </s>. That asymmetry is a property of the factorization, not of either implementation, and it means the two arms cannot be matched on it: a diffusion decode is told the width of its answer before its first forward pass.

Does it memorize?

At 4 epochs it should not, and unlike the 426-epoch case above it was checked rather than assumed — against the training stream, byte-verified, with real held-out Urdu as a control:

Training data

A frozen, decontaminated Urdu corpus of 85,362,688 unique tokens built for this project. The pipeline is ten stages — quality filtering, exact and near-duplicate removal, PII screening, held-out decontamination, and a script-aware split — and every stage carries its own decision record in the source repository.

Mixture, by tokens: 74.3% native Urdu, 20.4% Roman Urdu, 5.4% code-switched.

The tokenizer is a 16,384-piece SentencePiece model trained on this corpus (fingerprint 2855877c8ecd38c9). It ships with the weights, and the model cannot be used with any other one.

Licensing

The released weights are Apache-2.0. Urdu Wikipedia is CC-BY-SA-3.0, and whether share-alike reaches model weights is genuinely unsettled; this release takes the conventional position that weights are not a derivative work of training text, and discloses the input licence rather than relying on the reader not to ask. No raw training text is redistributed — the corpus ships as code, a manifest and checksums, so it can be rebuilt but not downloaded.

Limitations — read this part

These are measurements from this project, not boilerplate.

The Urdu was never read by a native speaker. Every judgement of fluency, grammaticality and sense in this project — including the samples above — was made by an LLM. A human evaluation was designed and specified and never ran: the three annotators it needed were never recruited. Treat every qualitative claim as unvalidated.

A known lexical defect is in the training data, disclosed rather than filtered. The Roman-Urdu source renders eight common Urdu words as fixed unrelated words — کرتے→baghaawat, بس→dehli, گھر→mamu — at 52–93% of their occurrences. That is 2.77% of rows after deduplication, inside the 20.4% of the corpus that is Roman Urdu. It was left in and stated, because filtering the eight confirmed cases cannot reach the unscreened tail without the native-speaker pass that never happened. Transliteration output is the worst affected.

Part of the held-out set is not decontaminated. The Roman-Urdu held-out split carries within-source deduplication only and never went through the containment pass; the native-Urdu and code-switched splits did. Separately, roughly 4.4% of the native-Urdu held-out split is estimated to also occur inside the FineWeb2 training pool. Both inflate held-out scores, and both do so for the two arms equally — so the AR-vs-diffusion comparison is largely robust to them, while the absolute bits-per-byte figures are optimistic.

One seed. Every number here comes from a single training run per arm. There is no estimate of seed-to-seed variance on this pair, so the gap between the two models cannot be separated from one initialization draw. (The diffusion objective was replicated across two seeds on a different, smaller corpus, where the two agreed to 0.29σ — that bounds the objective's stability, not this comparison.)

70M parameters. This is a small research model. It has no instruction tuning, no alignment, no safety filtering, and it will produce fluent-sounding false statements. It is trained on filtered web text and reproduces the biases of that text. It is not a product.

A pretraining coherence gate was recorded as failed. A 25M-parameter pilot was required to produce coherent Urdu before the full runs were funded, and it did not. The gate's kill criterion was "implementation bug — debug, do not scale"; no bug was found, and the gate's premise was judged wrong rather than the code — a 25M model on 7.4M tokens was judged below the scale at which coherent Urdu is reachable at all. That judgement was accepted with the risk named, and it is recorded here as unmet rather than quietly passed.

Provenance

Trained on one rented RTX 5090, 16 epochs, about $2; the released checkpoint is the 4-epoch rung of that run. The design was preregistered before any training — four falsifiable predictions and a deviation log recording every departure. Both are in the source repository, with a findings register of 71 numbered findings.

• Source, preregistration, corpus pipeline: https://github.com/AwaisAdilKhokhar/ravaan
• Matched diffusion model: https://huggingface.co/AwaisAdilKhokhar/ravaan-diff-70m