Model card: AnuLM-Base-400M (ckpt_multi36k.pt)
The three-language base model: pretrained from scratch on Hindi, English
and Python with the multi32k tokenizer, 36,000 steps at batch 8 × 512
(about 150M tokens), on one consumer GPU. It is the checkpoint the
translation and question-answering models were fine-tuned from, and the
one to continue Hindi or English prose with. Full log: docs/RESULTS.md
§22; the Hindi-only ladder that led to it is §1–21.
Licence: CC BY-SA 4.0. Hindi Wikipedia and Wikisource are CC BY-SA; C4 is ODC-BY; the Python slice is codeparrot-clean, de-duplicated GitHub Python with mixed licences. Not affiliated with Sarvam AI, AI4Bharat, BharatGen or the Government of India.
What it does
Continues text in the register it was given. A Hindi title followed by a blank line yields a Wikipedia- or Wikisource-style article; a narrative phrase yields narrative; a Python signature yields code. Val loss 4.441 at step 36,000 on the mixed held-out split. Everything factual in its output is invented; treat names, dates and numbers as fiction.
Data
| part | source | licence |
|---|---|---|
| Hindi | Hindi Wikipedia + Wikisource dumps (fetch_hindi.py, filter_prose.py) | CC BY-SA |
| English | C4 (allenai), a small slice (fetch_web.py) | ODC-BY |
| Python | codeparrot-clean (fetch_code.py) | mixed, GitHub |
Corpus and tokenizer build: experiments/build_multi.sh; training:
experiments/run_multi.sh. Both re-fetch everything from public sources;
no data is redistributed.
Architecture
The configuration the ablations in docs/RESULTS.md §12 and §16 converged
on: 20 layers, hidden 1,024, GQA with 16 query / 4 key-value heads and
QK-norm, RoPE θ = 1,000,000, sliding window 256 on layers 0–9, a dense
SwiGLU MLP on layer 0 and 24 routed experts of 192 with top-4 routing on
layers 1–19, aux-loss-free bias balancing (bias_update_rate 3e-3), no
shared expert, context 512, 32,768-entry vocabulary, untied embeddings.
397.7M parameters, 173.5M active per token. An independent
re-implementation of the Sarvam 30B design; docs/ARCHITECTURE.md §10
says which of Sarvam's choices held up at this scale.
Use
python sample.py --ckpt <this folder> --prompt "हिन्दी साहित्य
"
python serve.py --ckpt <this folder> # continue-the-text page
To fine-tune it, finetune.py --ckpt <this folder> --qa pairs.jsonl takes
question/answer JSONL; docs/DEVELOPING.md describes the format.
How to load
With transformers
The modelling code travels with the weights, so trust_remote_code=True is
required and there is nothing to clone:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("toonist/AnuLM-Base-400M", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("toonist/AnuLM-Base-400M")
ids = tok("\u092d\u093e\u0930\u0924 \u0915\u0940 \u0930\u093e\u091c\u0927\u093e\u0928\u0940", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=False)[0]))
Greedy output is identical to the AnuLM repository's own generate, cached or
not, and the tokenizer here agrees with bpe.py token for token; both are
pinned by tests in that repository.
Load it in float32 — which the config now asks for, so the line above is
enough. Do not force dtype=torch.bfloat16: this is a mixture-of-experts model
whose router keeps a per-expert bias, and rounding that bias to 16 bits changes
which experts fire. The output does not get slightly worse, it collapses into
repeated tokens. The big tensors are stored in bfloat16 and upcast on load; the
router bias and the norms are stored in float32 for this reason. For speed, use
torch.autocast over float32 weights, which is how every number in this card
was measured. The model loads the code from this repo at
trust_remote_code=True; pin a revision if you want that fixed.
Batches must be unpadded (one sequence at a time), and beam search is not supported.
With the AnuLM repository
Every script there takes this folder wherever it takes a .pt:
hf download toonist/AnuLM-Base-400M --local-dir AnuLM-Base-400M
python serve.py --ckpt AnuLM-Base-400M # web page at http://127.0.0.1:8000
python sample.py --ckpt AnuLM-Base-400M --prompt "\u092d\u093e\u0930\u0924 \u0915\u0940 \u0930\u093e\u091c\u0927\u093e\u0928\u0940"
from model import load_checkpoint, AnuLM
ck = load_checkpoint("AnuLM-Base-400M")
m = AnuLM(ck["cfg"]).eval(); m.load_state_dict(ck["model"])
Files: model.safetensors (397.7M parameters), tokenizer.multi32k.json for bpe.BPE.load,
and tokenizer.json in the tokenizers format for AutoTokenizer.