OneNMT v3b — Indic Multilingual Neural Machine Translation

OneNMT v3b is a 1.1B parameter multilingual seq2seq translation model supporting 36 × 36 language pairs across 35 Indian languages and English. It is based on a deep transformer_18_18 architecture (18 encoder + 18 decoder layers) trained with fairseq and converted to the HuggingFace MBart format.

⚠️ Tokenization Warning: OneNMT v3b uses a custom fairseq dictionary (fairseq_dict.json) for token ID mapping. Using AutoTokenizer or any standard MBart tokenizer will produce wrong IDs and garbled output. Always use the provided hf_inference.py or the translate_onemt() wrapper.

Architecture

Supported Languages

Installation

pip install -r requirements.txt

Quick Start

from hf_inference import translate_onemt, translate_batch

# Single sentence — English → Telugu
print(translate_onemt("Hello, how are you?", "eng", "tel"))
# హలో, మీరు ఎలా ఉన్నారు?

# Code-mixed Hindi → Telugu
print(translate_onemt("मुझे meeting attend करनी है।", "hin", "tel"))
# నాకు మీటింగ్‌కు హాజరు కావాలి.

# Telugu → English
print(translate_onemt("నమస్కారం, మీరు ఎలా ఉన్నారు?", "tel", "eng"))
# Hello, how are you?

# Tamil → Hindi
print(translate_onemt("இன்று வானிலை நன்றாக உள்ளது.", "tam", "hin"))
# आज मौसम अच्छा है।

# Batch translation
sentences = ["Good morning.", "Thank you.", "See you tomorrow."]
results   = translate_batch(sentences, sl="eng", tl="hin", batch_size=32)
for src, tgt in zip(sentences, results):
    print(f"{src}  →  {tgt}")

How Tokenization Works

Unlike standard MBart, OneNMT v3b uses a two-step tokenization pipeline:

source text
  → prepend language tag:   ###eng_Latn-to-tel_Telu### {text}
  → SentencePiece encoding  →  subword pieces
  → fairseq dict lookup     →  integer IDs   ← NOT SPM-native IDs
  → MBart encoder

The fairseq dictionary orders tokens by training frequency (descending), not SPM's internal ordering. This means the same subword piece maps to completely different IDs in the two systems — for example, T is ID 29 in the fairseq dict but ID 35376 via the HF tokenizer. Wrong IDs → wrong embeddings → garbled output.

The hf_inference.py script handles this correctly using onemtv3b_spm.model for segmentation and fairseq_dict.json for ID lookup.

Repository Files

tokenizer.json, tokenizer_config.json, special_tokens_map.json are present for HF Hub format compliance but must not be used for inference — they produce incorrect token IDs due to the SPM vs fairseq ordering mismatch described above.

Generation Parameters

Citation

@misc{mujadia2025bhashaversetranslationecosystem,
      title={BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages}, 
      author={Vandan Mujadia and Dipti Misra Sharma},
      year={2025},
      eprint={2412.04351},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.04351}, 
}

License

MIT