Ateso ↔ English (NLLB-200 fine-tune) — v3

Bidirectional machine translation between Ateso (teo, Eastern Nilotic, ~3 million Iteso speakers in eastern Uganda and western Kenya) and English.

Fine-tuned from facebook/nllb-200-distilled-600M, which contains no Eastern Nilotic language at all. The teo_Latn token in this model does not exist in base NLLB-200 — it was added and trained here.

Live demo: https://ateso.tesotunes.com

What v3 fixes

v1 of this model translated Ateso → English, but when asked for Ateso it produced fluent Luganda — an unrelated Bantu language.

The cause was a silent training bug worth documenting, because it is easy to hit and nearly invisible. The new teo_Latn embedding was initialised by copying the lug_Latn row, and the model was then wrapped in LoRA targeting only attention and feed-forward projections, with no modules_to_save. That froze the embedding table. The Luganda seed was made untrainable and never moved:

torch.equal(emb[teo_id], emb[lug_id])   # True
(emb[teo_id] - emb[lug_id]).abs().max() # 0.000e+00

Because NLLB ties lm_head to the input embedding, forcing forced_bos_token_id=teo_Latn was mathematically identical to forcing lug_Latn. Ateso → English worked because the encoder had learned Ateso; only the decoder side was never trained.

v3 changes three things:

1. Full fine-tune rather than LoRA, so embeddings actually train
2. Re-seeded teo_Latn away from Luganda, norm-matched to the ~9.4 typical of NLLB language tokens
3. Bidirectional training, so Ateso → English does not regress

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("Egonyu/ateso-english-nllb-v3")
model = AutoModelForSeq2SeqLM.from_pretrained("Egonyu/ateso-english-nllb-v3")

def translate(text, src, tgt):
    tok.src_lang = src
    x = tok(text, return_tensors="pt", truncation=True, max_length=128)
    y = model.generate(**x, forced_bos_token_id=tok.convert_tokens_to_ids(tgt),
                       max_length=128, num_beams=4)
    return tok.batch_decode(y, skip_special_tokens=True)[0]

translate("The children are going to school.", "eng_Latn", "teo_Latn")
# 'Elosete idwe isikolo.'

translate("Kotoma ageunet abu Edeke kosub akuj ka akwap.", "teo_Latn", "eng_Latn")
# 'In the beginning God created the heavens and the earth.'

forced_bos_token_id is required. Without it the model will not reliably select the target language.

Evaluation

Language identity (the metric that matters here)

† The one coincidence is Genesis 1:1, where the teo_Latn and lug_Latn outputs are identical because both are Ateso (ateso_lex 1.00) — the model now produces Ateso even when Luganda is requested. Distinctness was the right test while the failure mode was "emits Luganda"; it is uninformative once the model emits Ateso for either token.

Held-out probe outputs (not scripture):

The children are going to school.   -> Elosete idwe isikolo.
Good morning, how are you today?    -> Ejok, ikonakini ai?
Thank you very much for your help.  -> Kimalaenenete noi kanuka aiŋarakinikus.

Ateso → English did not regress:

Kotoma ageunet abu Edeke kosub akuj ka akwap.
  -> In the beginning God created the heavens and the earth.

Corpus scores

Validation split of the training corpus, 150 pairs per direction, beam size 4:

ateso_lex on out-of-domain everyday sentences (the probes above) is 0.842 — lower than the in-domain 0.957, as expected, and still far from Luganda's 0.000.

ateso_lex is the fraction of output words appearing in the Ateso lexicon of the training corpus. It exists because BLEU cannot distinguish "bad Ateso" from "fluent Luganda", which was precisely the v1 failure. For reference, v1's English→Ateso output scored 0.000; Luganda text scores 0.000 by construction.

Read these numbers with care

The validation split is shuffled at the pair-direction level, so a verse held out as English→Ateso very likely had its Ateso→English twin in training. These scores overstate generalisation. Some output is visibly memorised rather than translated:

hyp: But thou, O LORD, art high above for ever.
ref: But thou, LORD, art most high for evermore.

That is a model that has read this Bible many times, not one that has learned general Ateso. A clean held-out benchmark for Ateso does not exist yet; building one is a priority and an open invitation to collaborators.

Scores are also on in-domain biblical text. Expect materially worse performance on everyday speech (see Limitations). Treat BLEU 44.20 as flattering rather than false — the language-identity results above are the load-bearing evidence for what v3 actually fixed.

Training

• Base: Egonyu/ateso-english-nllb (itself a facebook/nllb-200-distilled-600M fine-tune), chosen over base NLLB so the encoder's existing Ateso competence was retained
• Data: 30,955 deduplicated Ateso–English verse pairs (ateso_1961 × english_kjv), used in both directions → 61,910 training rows
• Objective: full fine-tune, all 615.1M parameters trainable
• Optimizer: Adafactor, LR 1e-4, cosine schedule, 3% warmup, weight decay 0.01
• Batch: 4 × 8 gradient accumulation (effective 32), max length 128
• Precision: fp16 with gradient checkpointing
• Hardware: single 16 GB NVIDIA T4, ~15 hours

Preprocessing normalises curly quotes (“ ” ‘ ’), which NLLB's tokenizer otherwise maps to literal <unk>.

Note on memory

NLLB-200-distilled-600M is heavier than its parameter count suggests: 43% of its 615.1M parameters are the 256205 × 1024 embedding, and the logits tensor is batch × length × 256205. A full fine-tune with AdamW needs ~10.3 GB before any activations and OOMs on a 16 GB card. Adafactor reduces optimizer state from 4.58 GB to ~0.05 GB, which is what makes a T4 viable. Adafactor also needs a higher learning rate than Adam or it stalls.

Intended use

Translation between Ateso and English: language learning, accessibility, and research on low-resource Nilotic MT. It is a research prototype, not a production translation service.

Out of scope

• This is no longer a multilingual model. Full fine-tuning on Ateso/English has degraded NLLB's other 199 languages — forcing lug_Latn now also returns Ateso-like text. Do not use v3 as a general NLLB replacement.
• Anything safety-critical: medical, legal, financial, or emergency communication
• Commercial use — see Licence below
• Formal or published translation without review by a fluent Ateso speaker

Limitations and bias

Register. Training data is the Bible (1961 translation). The model writes archaic, scriptural Ateso and carries the domain's vocabulary and worldview. Everyday speech, modern vocabulary and conversational register are all weak.

Religious bias. A model trained solely on scripture will lean toward religious phrasing and may inject it where it does not belong.

Subword fragmentation. NLLB's tokenizer was built without Ateso. Ateso fragments into 2.42 subwords per word against English's 1.33, and only 7.4% of Ateso word types survive as a single token. This caps quality and inflates sequence length. Fixing it means extending the vocabulary, which is not yet justified at this data scale.

Dialect. Ateso has dialectal variation across Uganda and Kenya; this reflects whatever the 1961 translation used, without documenting it.

Data scale. ~31k pairs from one source and one domain is small for teaching a language the decoder had never generated.

Ateker roadmap

Base NLLB-200 contains no Ateker (Teso-Turkana) language: Turkana (tuv), Karamojong (kdj), Toposa (toq) and Nyangatom (nnj) are all absent, as are the neighbouring Lotuko-Maa languages including Maasai.

The intended path is one ISO language token per language sharing a single model — not a pooled ateker_Latn token, which would leave no way to select which language to emit and would recreate exactly the v1 failure inside the family. Now that teo_Latn is trained it becomes the natural seed for its siblings, which is the close relative Ateso itself never had.

Licence

cc-by-nc-4.0, inherited from facebook/nllb-200-distilled-600M.

NLLB-200 is released under CC-BY-NC-4.0, so this fine-tune is a derivative work under the same non-commercial terms. Commercial deployment is not permitted without separate arrangements with the upstream rights holders.

Corpus rights are a separate question — see the dataset card.

Citation

@misc{egonyu2026ateso,
  title  = {Ateso--English Neural Machine Translation},
  author = {Egonyu, Daniel},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Egonyu/ateso-english-nllb-v3}},
  note   = {Fine-tuned from NLLB-200-distilled-600M. Ateker Language Initiative}
}

Acknowledgements

Built on Meta AI's NLLB-200. Ateso scripture from the United Bible Societies (1961) and the Bible Society of Kenya (2020). Part of the Ateker Language Initiative.

Corrections from Ateso speakers via the live demo feed directly into future versions — the everyday-register data the Bible corpus cannot provide.