nllb-sango-finetuned-600m-v2-2 — Sango↔French translation, honestly benchmarked

Recommended use: Sango → French. On out-of-distribution text, this model beats the untrained NLLB-200 baseline in the Sango→French direction on two independent benchmarks: FLORES+ devtest (chrF 31.88 vs 31.37, BLEU 10.67 vs 9.84) and SIB-200 test (chrF 31.45 vs 30.20, BLEU 9.19 vs 7.65), beam=4. For French → Sango, the vanilla baseline currently remains stronger out-of-domain (−1.14 chrF on FLORES, −1.71 on SIB) — we recommend the baseline for that direction, and we publish the numbers that say so. Best practice: route by direction.

It is a published, openly benchmarked fine-tuned model (CC BY-NC 4.0, inherited from the base — not an open-source licence) that beats the NLLB baseline on out-of-distribution Sango→French — built by a native Sango speaker on free compute.

Why this model card is different

Our previous releases (v1, and a deleted v2) improved in-distribution scores while regressing below the baseline out-of-distribution — classic overfitting, published-then-caught. Since then, every MEYNG release passes an OOD gate before shipping: out-of-distribution chrF must meet or beat the baseline, or the model does not ship. This model passed that gate in its recommended direction across four consecutive training runs. The failing direction is documented, not hidden.

Benchmarks (beam=4, transformers <5.0)

FLORES+ devtest (canonical OOD ruler, 500 sentences):

SIB-200 test (204 sentences):

chrF is primary (BLEU is unreliable for Sango morphology; never quote bare BLEU). The SAG→FR advantage is consistent across both benchmarks (five consecutive gate passes across four training runs). Selection during training used SIB-200 train+validation only; neither test set above was ever used for selection. FLORES+ used under its dataset terms (openlanguagedata/flores_plus).

⚠️ A caveat on the SIB-200 FR→SAG numbers (added 2026-09-06)

The Sango side of the SIB-200 test data is orthographically inconsistent. Measured across the 204 sentences: only 8.3% of vowels carry a tone mark, and the marking follows three competing conventions at once — the same lemma appears as ti 854×, tî 3× and tï 2×; so 192× and sô 46×; tene 41× and tènè 21×. Sango tone is lexically contrastive, so a system that spells a word correctly in a convention the reference happens not to use is scored as wrong.

What this does and does not affect:

• SAG→FR — unaffected. That direction is scored against the French references, so the Sango-side inconsistency changes how hard the input is, not how the output is scored. The +1.25 chrF PASS stands.
• FR→SAG — the delta stands; the absolute numbers do not measure tone quality. Both systems are scored against the same reference, so the −1.71 chrF comparison remains valid and we continue to recommend the baseline for this direction. But 34.32 and 36.03 are each depressed for any system that marks tone at all, and neither should be quoted as evidence about orthographic correctness.

We have not measured the FLORES+ Sango reference for the same property, so no claim is made about it here.

The data story (read this — it explains everything)

The public NLLB web-mined French–Sango corpus advertises 588,039 quality-filtered pairs. A forensic cleaning pass found:

• 460,437 rows (78%) were duplicate Sango targets — the same sentences repeated;
• 100,683 rows carried leading verse numbers, 29,939 speaker labels, 12,095 scripture references (JW.org-heavy sourcing) — all repaired;
• honest unique corpus after cleaning: 45,451 pairs (7.7%).

Earlier models trained on the raw corpus learned these artifacts (verse-number hallucination, register memorization) — the direct cause of their OOD regression. This model trains on the cleaned 45,451 pairs only. The cleaning pipeline is deterministic and reproducible.

What we tried (five-experiment log, all OOD-gated)

Negative results included on purpose: small high-quality vocabulary injection and corpus volume did not improve OOD generation. The consistent early-peak/dip training curve indicates the remaining FR→SAG gap is a register-diversity limit of the available corpus, not a training recipe problem. Closing it requires new diverse Sango text — ongoing work.

Training details

• Base: facebook/nllb-200-distilled-600M (615M), correct per-direction NLLB language-token conditioning (asserted at train time; requires transformers>=4.40,<5.0 — v5 breaks the NLLB tokenizer prefix).
• LoRA: r=16, α=32, dropout 0.05, attention projections (q,k,v,out) — 4.72M trainable (0.761%).
• Data: 45,451 cleaned unique FR–Sango pairs, bidirectional (90,902 rows), fp16, effective batch 64, lr 1e-4.
• Checkpoint selection on out-of-distribution chrF (SIB-200 train+validation), evaluated every 200 steps; best checkpoint (step 200) merged into standalone weights — no peft needed at inference.
• Hardware: single free Kaggle T4. Total training cost of the entire 5-run program: $0.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer  # transformers>=4.40,<5.0

model_id = "MEYNG/nllb-sango-finetuned-600m-v2-2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

def translate(text, src="sag_Latn", tgt="fra_Latn"):  # recommended direction
    tok.src_lang = src
    enc = tok(text, return_tensors="pt")
    out = model.generate(**enc,
                         forced_bos_token_id=tok.convert_tokens_to_ids(tgt),
                         max_length=160, num_beams=4)
    return tok.batch_decode(out, skip_special_tokens=True)[0]

print(translate("Mbi yeke gue na da ti nganga ti kobela."))

For French→Sango, use facebook/nllb-200-distilled-600M directly (per our own benchmark above).

Limitations

• FR→SAG trails the baseline OOD (−1.14 chrF on FLORES, −1.71 on SIB): use the baseline for that direction.
• Domain skew: training data is predominantly religious/NGO register web text; performance on conversational or technical Sango is untested beyond SIB-200.
• "Verified" in our vocabulary dataset means pipeline-verified; native-rubric human evaluation is in progress.
• License: CC-BY-NC-4.0 (inherited from the NLLB base model) — non-commercial use.

About MEYNG

MEYNG builds language AI infrastructure for under-resourced African languages, starting with Sango, the national language of the Central African Republic. Native-speaker verification, out-of-distribution gates before every release, failures documented. Every public number we cite, with its definition, source and date: https://sangoai.app/metrics.json · https://www.meyng.com/en/metrics. Datasets, methods, and honest benchmarks: https://huggingface.co/MEYNG · https://meyng.com

If you're building AI for a language the world hasn't gotten around to yet — the dataset is open, the methodology is documented, the door is open.