Chandohasam pilot120m: a 120M masked diffusion language model for Telugu

pilot120m is a 120M-parameter masked diffusion language model (MDLM; Sahoo et al., 2024) for Telugu, trained from scratch on 4.98B tokens of web, news and encyclopedic text. Instead of writing left to right, it fills a canvas of 512 <mask> tokens in any order, seeing the whole canvas at every step. Its tokenizer gives one token per akshara (orthographic syllable), so every token is a unit of Telugu metre.

It is stage 1 of Chandohasam, a project on generating Telugu metrical poetry (padyams), whose metres fix the weight of every syllable, a rhyme in the second syllable of every line (prāsa) and a caesura agreement (yati). This stage teaches general Telugu; a later stage specialises the model to verse, decoded under exact metrical constraints. It has seen no poetry: documents that overlap a large collection of known Telugu poems were removed from its training data.

Contents: Quick start · Intended use and limitations · Architecture · Tokenizer · Diffusion objective · Sampling · Training data · Training procedure · Evaluation · Growth compatibility · Files · License · Citation · References

Quick start

The model is a custom architecture; the code to load and sample it is in this repository (chandohasam_mdlm/). It needs Python ≥ 3.10, PyTorch ≥ 2.4 and safetensors.

pip install torch safetensors huggingface_hub

import sys
from huggingface_hub import snapshot_download

repo = snapshot_download("samvaran/chandohasam")
sys.path.insert(0, repo)
from chandohasam_mdlm import load, generate

model, tok = load(repo, device="cuda")           # or device="cpu"
for text in generate(model, tok, n=2, steps=512, temperature=0.9, seed=0):
    print(text, "\n---")

# fix the start of every canvas
print(generate(model, tok, n=1, prefix="తెలుగు భాష", seed=1)[0])

Or from a clone of the repository: python generate.py --n 4 --steps 512 --temperature 0.9.

generate starts from 512 <mask> tokens and unmasks them with the ancestral sampler. 512 steps at temperature 0.9 gave the most fluent samples in the evaluation (Sampling). On the training GPU (RTX 5050 Laptop), 64 canvases at 512 steps take about 12 minutes.

The denoiser can also be used directly: model(x) maps token ids x (batch × ≤ 512, with tok.special_to_id["<mask>"] at the positions to predict) to logits over the padded vocabulary of 45,696. Before taking a softmax, add invalid_bias(...) from chandohasam_mdlm, which sets <mask>, <pad>, <unk> and the 105 padding rows to −∞, as in training.

Intended use and limitations

Intended use. Research on masked diffusion language models for Telugu, and a starting point for continued pretraining or fine-tuning, in particular for constrained generation where each position is an akshara: metrical verse, infilling, and decoding steered by a rule engine.

It is a base model. It is not instruction-tuned, chat-tuned or safety-tuned, and it is not a source of facts. Its unconditional samples are fluent phrases in a web, news or blog register without a coherent thread across sentences. At the recommended setting, an independent judge (Gemma-3-1B) rates them at a perplexity of 45.8, against 16.9 for real Telugu text, and 86.4% of their words are real words, against 88.7% in real text.

Known limitations.

• The heavily masked regime is weak. With 85% of the tokens masked, the model predicts 31–32% of them correctly. Generation starts from a fully masked canvas, so its first steps run in this regime, and more denoising steps barely help (Evaluation).
• Confidence-ordered unmasking collapses into repeated tokens for unconditional generation. Use the ancestral sampler.
• The data is web and news text. Generated text can mention real people, places and events and say false things about them, and it can reproduce biases and offensive language present in web text. Nothing beyond the cleaning described below was done to filter content.
• Telugu only. Other scripts are kept losslessly as bytes by the tokenizer but are not modelled.
• No poetry. Poems were deliberately removed from the training data, so the model knows the language of verse only as far as it appears in prose.
• The canvas is fixed at 512 tokens. Training windows are 512 tokens cut from concatenated documents, so a canvas may contain the end of one document and the start of the next, separated by <eos> <bos>.

Architecture

Hyperparameters

Design choices

Weight tensors

All 74 tensors (i = 0…11):

Each block holds 7,079,424 parameters, so the 12 blocks hold 84,953,088; the final norm (768) and the tied embedding (35,094,528) make up the rest.

Tokenizer

• Vocabulary. telugu_alldomain.tokenizer.json (sha256 200c814d84cc1b88bccdea3e4131a4c2fee296d050587e5d3da99f22d73d30fc), 45,591 tokens: 44,835 whole aksharas, 485 byte-level BPE pieces, 256 bytes, 10 Telugu digits and 5 specials: <pad> = 0, <unk> = 1, <bos> = 2, <eos> = 3, <mask> = 4.
• One token per akshara. Text is split into aksharas by aksharanusarika (included, MIT), the same splitter as the Chandohasam metre engine, so token boundaries are metrical boundaries. The arasunna ఁ stays on its akshara and zero-width non-joiners are removed.
• No out-of-vocabulary input. An akshara without its own token falls back to byte-level BPE. On unseen Sangraha news, 99.94% of syllables are single tokens.
• Lossless. decode(encode(text)) returns the normalised text.

Diffusion objective

Sampling

• Ancestral sampler (chandohasam_mdlm/sampling.py), MDLM's: going from t to s < t, each still-masked position is revealed with probability (t − s)/t, its token drawn from the model. Because the model has no time input, a step that reveals nothing reuses the previous forward pass.
• Confidence order (MaskGIT/LLaDA style) is also implemented: each step reveals the masked positions whose sampled token is most probable. For unconditional generation it collapses into repetition.
• Categorical draws use float64 Gumbel noise. Float32 noise quietly lowers the sampling temperature and flatters sample quality (Zheng et al., 2025).
• Recommended: ancestral, 512 steps, temperature 0.9.

Training data

Sources

Cleaning, deduplication and decontamination

• Normalisation. NFC, the tokenizer's normalisation, control characters removed, whitespace collapsed, literal \n sequences turned into newlines.
• Exact duplicates and boilerplate. Exact-duplicate documents are dropped, and so are boilerplate lines (any line found in 10 or more distinct documents; 90,446 such lines). IndicCorp paragraphs that already occur inside a Wikipedia or Sangraha document (5,518,447) are dropped.
• Cleaning. URLs and e-mail addresses are cut out of their lines. A line is dropped if under 50% of its letters are Telugu. A document is dropped if it has fewer than 40 Telugu letters, if under 80% of its letters are Telugu, or if more than 30% of its lines are repeats. After tokenization, a document is dropped if more than 2% of its Telugu pieces have no whole-akshara token (garbled text), or if under half of its tokens carry text rather than whitespace, ASCII punctuation or digits (tables and number lists).
• Near duplicates. MinHash (64 permutations over 8-token shingles), clustered at an estimated Jaccard similarity ≥ 0.8 (8 bands × 8 rows), one document kept per cluster; for Wikipedia and Sangraha.
• Decontamination. Any document that shares a 20-token window with a large collection of known Telugu poems is dropped, so that the later poetry stage is not contaminated.
• Splits. Documents go to train, validation (0.5%) and test (0.5%) by a stable hash. Each document is stored as <bos> … <eos>.

Documents at each stage:

For IndicCorp, "duplicate" includes the paragraphs already present in another source, and most cleaning rejects are paragraphs too short to keep.

Tokens, including <bos> and <eos>:

Mixture

Each training example is 512 consecutive tokens from a random offset in the chosen source's train split, so a window may span several documents (MDLM's "wrapped" setting).

Training procedure

The run never diverged, never ran out of memory and never skipped an update. It rode through two mains-power cuts on battery without losing work.

Evaluation

Likelihood

NELBO in nats per token (lower is better) of the released weights, on 512 fixed canvases per source with fixed mask rates and masks. exp(NELBO) is an upper bound on the per-token perplexity.

The test split was untouched until this evaluation and is no harder than validation. Wikipedia scores low because, beyond its first 128 canvases, its held-out text is dominated by templated village articles, which are nearly predictable; the per-source and mix-weighted values are the ones to compare.

During training, the validation NELBO on 128 fixed canvases per source fell from 2.921 at step 1,000 to 1.747 at step 76,000, and was still falling slowly at the end.

Masked-token accuracy

Top-1 accuracy on masked tokens (test split): 87.9% with 15% of the tokens masked, 69.1% with 50% and 31.4% with 85% (validation: 89.4%, 68.8%, 32.3%).

Sample quality and sampler settings

64 canvases of 512 tokens per setting, generated from an all-mask canvas with the released weights. Judge perplexity is the perplexity of the samples under an independent model, google/gemma-3-1b-pt (Gemma Team, 2025); on real validation text it is 16.9, and 64.9 when the words of that text are shuffled. Real words: the share of generated words that occur in validation text (88.7% for real text).

• The model, not the sampler, is the limit. Four times as many steps (256 → 1,024) bring judge perplexity only from 71.5 to 64.9, the level of word-shuffled real text.
• Confidence order collapses. It fills the canvas with a few repeated tokens; the judge scores that as nearly perfect, which token entropy and repeated 4-grams expose.
• Temperature 0.9 gives the most fluent samples (45.8, 86.4% real words), at some cost in diversity.

An unedited excerpt from a sample at step 76,000 (training sampler: ancestral, 256 steps, temperature 1.0):

వంతెన, వేలం సీజన్ లో వారి నిర్ణయాలు తీసుకుంటుంటారు. వారు పర్యావరణ నిర్మాణం లేదని చేయాలి, ఇది ప్రైవేట్ గదులు, చాలా గది వాతావరణం ఒక సౌకర్యం ఆస్వాదించడం

Growth compatibility

The model is shaped so that larger models can be initialised from it function-preservingly: the grown model starts out computing exactly what this one does. Every stage keeps the head size (64), the RoPE base, the tokenizer and the 512-token canvas, and keeps the SwiGLU width at 8/3 × d_model; each width is a whole multiple of the previous one. Width grows by cloning (HyperCloning; Samragh et al., 2024), depth by inserting blocks whose output projections start at zero.

Files in this repository

License

The weights and code in this repository are released under the MIT License. aksharanusarika.py is © 2025 Aksharanusarika Contributors, also MIT (chandohasam_mdlm/tokenizer/LICENSE.aksharanusarika). The training data has its own terms: Sangraha (CC BY 4.0), Telugu Wikipedia (CC BY-SA 3.0 and GFDL) and IndicCorp v2 (no license stated on its dataset card).

Citation

@misc{chandohasam_pilot120m_2026,
  title        = {Chandohasam pilot120m: a 120M-parameter masked diffusion language model for Telugu},
  author       = {Rallabandi, Samvaran Kashyap and Emani, Mahesh and Salopanthula, Radhe Shyam},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/samvaran/chandohasam}}
}

References

• Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J. T., Rush, A. and Kuleshov, V. (2024). Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems 37.
• Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z. and Li, C. (2025). Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. International Conference on Learning Representations.
• Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J. and Zhang, Q. (2025). Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling. International Conference on Learning Representations.
• Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R. and Li, C. (2025). Large Language Diffusion Models. Advances in Neural Information Processing Systems 38.
• Samragh, M., Mirzadeh, I., Alizadeh Vahid, K., Faghri, F., Cho, M., Nabi, M., Naik, D. and Farajtabar, M. (2024). Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization. Proceedings of the 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, PMLR 262.
• Khan, M. S. U. R., Mehta, P., Sankar, A., Kumaravelan, U., Doddapaneni, S., B, S., G, V., Jain, S., Kunchukuttan, A., Kumar, P., Dabre, R. and Khapra, M. M. (2024). IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages. Proceedings of ACL 2024.
• Doddapaneni, S., Aralikatte, R., Ramesh, G., Goyal, S., Khapra, M. M., Kunchukuttan, A. and Kumar, P. (2023). Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. Proceedings of ACL 2023.
• Gemma Team (2025). Gemma 3 Technical Report. arXiv:2503.19786.