NLLB Russian ↔ Ruska Romani (Merged Model)
Full, ready-to-use translation model for Russian ↔ Ruska Romani, fine-tuned from facebook/nllb-200-distilled-600M using LoRA, then merged into the base weights. No PEFT dependency or merge step required — load and run directly.
Ruska Romani is not natively supported by NLLB-200. A custom language
code, ruk_Cyrl, was added to the tokenizer to represent it, with its
embedding warm-started from rom_Latn (Vlax Romani, the closest existing
NLLB language) before fine-tuning.
If you'd prefer the lightweight LoRA adapter instead (e.g. to compose
with other adapters, or minimize download size), see the companion repo:
midwestcyr/nllb-200-distilled-600m-rus-ruskaromani-lora.
Model details
- Base model: facebook/nllb-200-distilled-600M
- Fine-tuning method: LoRA (attention and feed-forward projections), merged into the base model weights after training
- New language code:
ruk_Cyrl(Ruska Romani, Cyrillic script) - Translation direction trained:
rus_Cyrl ↔ ruk_Cyrl - Training data: parallel Russian / Ruska Romani sentence pairs extracted from a JSON corpus, split 95% train / 5% validation
- Evaluation metric: SacreBLEU on the held-out validation split
Data
Ruska Romani is the dialect of the Romani language attributed to Ruska Roma, the largest subgroup of Romani people in Russia.
Training and evaluation data consists of parallel Russian / Ruska Romani sentence pairs from the Russian National Corpus. The data curation process is described in The Parallel Corpus of Russian and Ruska Romani Languages. Please cite that paper in any publication using this model or the underlying corpus.
Training
The model was fine-tuned using the Hugging Face Seq2SeqTrainer with LoRA adapters on parallel Russian ↔ Ruska Romani sentence pairs.
Training configuration
| Parameter | Value |
|---|---|
| Epochs | 8 |
| Per-device train batch size | 4 |
| Per-device evaluation batch size | 4 |
| Gradient accumulation steps | 2 |
| Effective training batch size | 8 |
| Learning rate | 3e-4 |
| Mixed precision | FP16 |
| Evaluation strategy | Every epoch |
| Checkpoint saving | Every epoch |
| Logging frequency | Every 50 steps |
| Generation during evaluation | Enabled |
| Maximum generation length | 256 |
| Best model selection | Highest validation SacreBLEU |
| Trainer | Hugging Face Seq2SeqTrainer |
Training results
| Metric | Value |
|---|---|
| Total epochs completed | 8 |
| Total optimization steps | 7,432 |
| Best validation SacreBLEU | 22.93 |
| Best model epoch | 8 |
The best-performing checkpoint was automatically restored at the end of training (load_best_model_at_end=True) using the validation SacreBLEU score as the model selection criterion.
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch
MODEL_REPO = "midwestcyr/nllb-200-distilled-600m-rus-ruskaromani"
device = torch.device("cuda" if torch.cuda.is_available() else
"mps" if torch.backends.mps.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(MODEL_REPO)
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_REPO).to(device)
model.eval()
def translate(text, src_lang, tgt_lang, max_length=128, num_beams=5):
new_lang_id = tokenizer.convert_tokens_to_ids(tgt_lang)
tokenizer.src_lang = src_lang
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=max_length).to(device)
generated = model.generate(
**inputs,
forced_bos_token_id=new_lang_id,
max_length=max_length,
num_beams=num_beams,
)
return tokenizer.batch_decode(generated, skip_special_tokens=True)[0]
# Russian -> Ruska Romani
src_lang = "rus_Cyrl"
tgt_lang = "ruk_Cyrl"
print(translate("Владимир разорвал их, не читая.", src_lang, tgt_lang))
### Output: Владимиро розрискирдя лэн на гины.
# Ruska Romani -> Russian
src_lang = "ruk_Cyrl"
tgt_lang = "rus_Cyrl"
print(translate("Владимиро розрискирдя лэн на гины.", src_lang, tgt_lang))
### Output: Владимир разорвал их, не читая.
Batch translation
def translate_batch(texts, max_length=128, num_beams=5, batch_size=16):
tokenizer.src_lang = "rus_Cyrl"
results = []
for i in range(0, len(texts), batch_size):
batch = texts[i:i + batch_size]
inputs = tokenizer(batch, return_tensors="pt", padding=True,
truncation=True, max_length=max_length).to(device)
generated = model.generate(
**inputs,
forced_bos_token_id=new_lang_id,
max_length=max_length,
num_beams=num_beams,
)
results.extend(tokenizer.batch_decode(generated, skip_special_tokens=True))
return results
Limitations
- Low-resource pair: dataset size is modest; translation quality on out-of-domain or long/complex sentences may be inconsistent.
- Custom language code:
ruk_Cyrlis not a standard NLLB/FLORES code and won't be recognized by other NLLB-based tools without the tokenizer shipped in this repo.
Citation / acknowledgment
Ruska Romani and Russian Parallel Corpus
@inproceedings{koncha-etal-2024-parallel,
title = "The Parallel Corpus of {R}ussian and Ruska {R}omani Languages",
author = "Koncha, Kirill and
Kukanova, Abina and
Tatiana, Kazakova and
Rozovskaya, Gloria",
editor = "Serikov, Oleg and
Voloshina, Ekaterina and
Postnikova, Anna and
Muradoglu, Saliha and
Le Ferrand, Eric and
Klyachko, Elena and
Vylomova, Ekaterina and
Shavrina, Tatiana and
Tyers, Francis",
booktitle = "Proceedings of the Third Workshop on NLP Applications to Field Linguistics",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.fieldmatters-1.1/",
doi = "10.18653/v1/2024.fieldmatters-1.1",
pages = "1--5"
}
Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model
@inproceedings{koishekenov-etal-2023-memory,
title = "Memory-efficient {NLLB}-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model",
author = "Koishekenov, Yeskendir and
Berard, Alexandre and
Nikoulina, Vassilina",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-long.198/",
doi = "10.18653/v1/2023.acl-long.198",
pages = "3567--3585",
abstract = "The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs. It contains 54.5B parameters and requires at least four 32GB GPUs just for inference. In this work, we propose a pruning method that enables the removal of up to 80{\%} of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics can identify language-specific experts."
}