Indic Translation Model — 55 Indian Languages

A fine-tuned translation model supporting 55 Indian languages including 33 low-resource languages. Translates bidirectionally between English and any supported Indic language.

Key Features

• 55 Indian languages — covers both major and low-resource regional languages
• Bidirectional — English to Indic and Indic to English
• 1.3B parameter encoder-decoder model, ready to use out of the box
• Fast inference — supports CTranslate2 int8 quantization for production deployments
• Extended vocabulary — 256,243 tokens with 33 newly added language codes

Supported Languages

How to Use

Quick Start (Transformers)

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

# Load model
model = AutoModelForSeq2SeqLM.from_pretrained("Anonym-050326/nirukti-translate-1.3b")
tokenizer = AutoTokenizer.from_pretrained("Anonym-050326/nirukti-translate-1.3b")

# Translate English to Hindi
tokenizer.src_lang = "eng_Latn"
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
translated = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("hin_Deva"),
    max_new_tokens=128,
)
print(tokenizer.decode(translated[0], skip_special_tokens=True))

CTranslate2 (Fastest — Recommended for Production)

import ctranslate2
from transformers import AutoTokenizer

translator = ctranslate2.Translator("ct2-int8", device="cuda", compute_type="int8_float16")
tokenizer = AutoTokenizer.from_pretrained("Anonym-050326/nirukti-translate-1.3b")
tokenizer.src_lang = "eng_Latn"

text = "Hello, how are you?"
encoded = tokenizer(text, return_tensors=None, max_length=256, truncation=True)
tokens = tokenizer.convert_ids_to_tokens(encoded["input_ids"])

result = translator.translate_batch([tokens], target_prefix=[["hin_Deva"]], beam_size=5)
output_tokens = result[0].hypotheses[0][1:]  # skip language token
output_ids = tokenizer.convert_tokens_to_ids(output_tokens)
print(tokenizer.decode(output_ids, skip_special_tokens=True))

Translating Between Any Language Pair

Set tokenizer.src_lang to the source language code and use the target code as forced_bos_token_id (Transformers) or target_prefix (CTranslate2).

# Tamil to English
tokenizer.src_lang = "tam_Taml"
# forced_bos_token_id=tokenizer.convert_tokens_to_ids("eng_Latn")

# English to Marwari
tokenizer.src_lang = "eng_Latn"
# forced_bos_token_id=tokenizer.convert_tokens_to_ids("mwr_Deva")

Testing Scripts

This repository includes evaluation and testing scripts in the testing_scripts/ directory:

Running Evaluation

# 1. Convert to CTranslate2 int8 for fast inference
ct2-opennmt-py-converter --model_path Anonym-050326/nirukti-translate-1.3b --output_dir ct2-int8 --quantization int8

# 2. Download benchmark datasets
python testing_scripts/download_datasets.py --output-dir eval_data

# 3. Run multi-GPU evaluation
python testing_scripts/run_evaluation.py \
    --model ct2-int8 \
    --tokenizer Anonym-050326/nirukti-translate-1.3b \
    --manifest eval_data/manifest.json \
    --output-dir results \
    --num-gpus 4 --batch-size 64 --beam-size 5

Training Details

Architecture

• Base: 1.3B parameter encoder-decoder translation model
• Fine-tuning: LoRA (rank 32, alpha 64) on attention + FFN layers, merged into base weights
• Trainable parameters: ~0.75% of total during training
• 2-stage training: embedding divergence followed by LoRA fine-tuning

Hyperparameters

Training Data

Trained on a curated combination of parallel corpora covering all 55 languages, with hash-based deduplication and quality filtering. Low-resource language pairs are upsampled using temperature-based balancing to ensure adequate representation.

Compute

• Training: 2x NVIDIA A100-SXM4-80GB, PyTorch DDP
• Inference: CTranslate2 int8 quantization

Limitations

• Low-resource languages (33 newly added) have less training data and lower translation quality than high-resource languages
• Performance varies by script — languages with multiple script variants may show different quality levels
• Training data is primarily from news, government, and religious domains; informal or conversational text may translate differently
• Very short sentences (1-5 words) are harder for the model due to limited context