MoVoC_MT: From-Scratch MarianMT for English-Amharic-Tigrinya, Zero-Shot Tigre
Downstream MT evaluation of MoVoC_Tok (a 120k-vocab shared Ge'ez-script + English SentencePiece Unigram tokenizer, built for the MoVoC project, arXiv:2509.08812). A from-scratch MarianMT model -- matching the paper's own architecture exactly (6+6 layers, 8 heads, d_model=512, ffn=2048, Swish activation, shared embeddings, static position embeddings) -- trained bidirectionally on English-Amharic and English-Tigrinya, then evaluated zero-shot on English-Tigre (a third Ge'ez-script language never seen during training).
Code: github.com/hailaykidu/MoVoC_MT
Training
- From-scratch (random init), NOT a fine-tune of any pretrained checkpoint -- no pretrained MarianMT model has a MoVoC_Tok-compatible (120k) vocabulary
- 106,104,832 parameters
- Bidirectional:
>>amh<</>>tir<</>>eng<<direction tags - 4,437,752 examples/epoch (2,157,134 Amharic-derived + 2,280,618 Tigrinya-derived), 3 epochs, 416,040 total steps
- Batch size 32, peak LR 5e-05, linear decay, 500 warmup steps, fp16
- Real runtime: 7h 25m 41s (SLURM job 52623, COMPLETED); throughput 502.8 samples/sec
- Per-epoch training loss: 3.6062 (epoch 1) -> 3.2935 (epoch 2) -> 3.1292 (epoch 3)
- Gradient-norm range: 1.33-5.61 (finite steps); 2 isolated
infevents (fp16 loss-scaler overflow, confirmed benign -- loss/eval continued improving smoothly around both)
Results (real, not adjusted)
Amharic / Tigrinya (2,000 held-out dev pairs each, beam search num_beams=4)
| Direction | BLEU | chrF |
|---|---|---|
| en -> am | 24.699 | 33.655 |
| am -> en | 20.485 | 45.554 |
| en -> ti | 20.556 | 18.634 |
| ti -> en | 10.571 | 31.945 |
Tigre, zero-shot (43 of 45 Tatoeba pairs; Tigre never in training data)
| Direction | BLEU | chrF |
|---|---|---|
| en -> tig | 11.713 | 19.405 |
| tig -> en | 17.628 | 32.187 |
X->English is consistently stronger than English->X in every language
pair tested, including zero-shot Tigre. tig->en zero-shot transfer is
genuinely reasonable (e.g. a source meaning "Technology has failed to ease
the conflict between man and nature" produces "Technology can't stop the
conflict between humans and nature" -- capturing the real meaning despite
zero Tigre training exposure). en->tig, by contrast, shows real
repetition-loop degeneration under beam search (e.g. "his stomach is weak"
produces a single Ge'ez word looping eight times) -- reported as observed, not
smoothed over. 43-45 pairs is a small qualitative check, not a
statistically robust benchmark; no larger English-Tigre parallel dataset
exists anywhere to draw a stronger conclusion from.
Usage
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained("Hailay/movoc-mt-en-am-ti")
tokenizer = AutoTokenizer.from_pretrained("Hailay/movoc-mt-en-am-ti")
text = ">>amh<< The weather is nice today."
inputs = tokenizer(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64, num_beams=4)
print(tokenizer.decode(out[0], skip_special_tokens=True))
Direction tags: >>amh<< (target Amharic), >>tir<< (target Tigrinya --
also used for zero-shot Tigre, the closest trained relative), >>eng<<
(target English).
Reproducibility
Partially reproducible. Data splitting/shuffling is seeded
(SEED=42) and deterministic, but exact metric recovery is not
guaranteed:
- Model weight initialization is not seeded -- the reported numbers correspond to one unrepeatable random init.
- Training uses fp16 with no forced-determinism flags (cuDNN algorithm selection / reduction order can vary run to run).
- No dependency lockfile is published; results were produced with
transformers==4.57.6,torch==2.9.0+cu128,datasets==5.0.0,tokenizers==0.22.2,sacrebleu==2.6.0. - The tokenizer and training corpora are referenced from external, unversioned paths, not vendored into the training repo.
Full details and code: github.com/hailaykidu/MoVoC_MT.
Limitations
- Amharic corpus capped at 1.5M raw NLLB lines (of 16.1M available), to balance against Tigrinya's corpus size.
- Batch size (32) deviates from the paper's original run (8) for wall-clock reasons on this larger, bidirectional, multi-pair corpus.
- Tigre zero-shot eval is only 43-45 pairs -- a small qualitative check, not a robust benchmark.
- En->X translation is weaker than X->En for every pair tested, and en->tig zero-shot shows real repetition-loop degeneration.
Installation
pip install transformers torch sentencepiece
Citation
If you use this artifact, please cite:
@inproceedings{teklehaymanot2025movoc,
title = {MoVoC: Morphology-Aware Subword Construction for Ge'ez Script Languages},
author = {Teklehaymanot, Hailay Kidu and Fazlija, Dren and Nejdl, Wolfgang},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
year = {2025},
url = {https://arxiv.org/abs/2509.08812}
}