opus-mt-en-ur-vocational
Fine-tuned version of Helsinki-NLP/opus-mt-en-ur on the VocabUR corpus for domain-adaptive English→Urdu translation of vocational trade instruction.
Model Description
The base opus-mt-en-ur model was trained on OPUS corpora (primarily religious texts, legal documents, and subtitles) and performs poorly on trade-specific language out of the box (BLEU 3.08 on vocational held-out set). Fine-tuning on VocabUR raises it to 20.83 BLEU, a ×6.8 relative improvement.
A companion NLLB-based model (mujtabaumarr/nllb-600M-vocational-urdu) achieves higher absolute performance (49.80 BLEU) on the same held-out set; that model is recommended for production use. This model is provided for research comparison.
Training Data
VocabUR — 3,442 training segments from ~32 hours of English vocational training video across three trade domains: electrical work, carpentry/woodworking, and plumbing. Human-validated translations by native Urdu speakers with trade knowledge. Transliteration of technical terms (e.g. stringers → سٹرنگرز, PEX → پی ای ایکس) is treated as the correct domain norm.
Evaluation
Evaluated on a video-independent held-out set (56 segments from 4 videos excluded entirely from
the training pipeline). BLEU computed with the flores200 tokenizer (sacrebleu
nrefs:1|case:mixed|eff:no|tok:flores200|smooth:exp|version:2.6.0).
| Model | BLEU (flores200) | chrF |
|---|---|---|
| opus-mt-en-ur zero-shot | 3.08 | 19.30 |
| opus-mt-en-ur fine-tuned (this model) | 20.83 | 42.91 |
Per-Domain BLEU (internal test split, n=354)
| Domain | Zero-shot | Fine-tuned | Gain |
|---|---|---|---|
| Carpentry | 2.9 | 19.9 | +17.0 |
| Plumbing | 7.0 | 26.5 | +19.5 |
| Electrical | 4.4 | 22.5 | +18.1 |
Training Configuration
| Hyperparameter | Value |
|---|---|
| Base model | Helsinki-NLP/opus-mt-en-ur |
| Epochs | 5 |
| Learning rate | 5e-5 |
| Effective batch size | 16 |
| Max sequence length | 256 tokens |
| Precision | fp16 mixed |
| Hardware | NVIDIA T4 16GB |
| Training time | ~50 minutes |
| Best checkpoint | Epoch 5 (val loss 0.842) |
Usage
from transformers import MarianMTModel, MarianTokenizer
model_name = "mujtabaumarr/opus-mt-en-ur-vocational"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)
text = ["Install the shutoff valve before connecting the PEX line."]
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
translated = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.batch_decode(translated, skip_special_tokens=True))
Citation
@inproceedings{umar2025vocabur,
title = {{VocabUR}: Domain-Adaptive {English--Urdu} Translation for Vocational Education},
author = {Umar, Mujtaba},
booktitle = {Proceedings of [Conference Name]},
year = {2025}
}
Limitations
- Evaluated on a small held-out set (56 segments); headline metrics carry wide confidence intervals.
- Single-reference BLEU/chrF may penalize valid transliteration variants.
- Source videos are North American English; South Asian trade vocabulary may differ.
- Not a substitute for certified trade training or expert supervision.