ebible_m2o-nllb600m-control-ilo
NLLB-200-distilled-600M fine-tuned to translate Bible verses into ilo from 4 related source languages. Part of a series testing whether a pretrained multilingual model can draft Old Testament books for a language it has only seen the New Testament of — the practical "no OT exists yet" scenario. Project: reproduction and extension of Sami Liedes' 2018 closed-text Bible translation experiment.
Training used New Testament verses only (ilo target, whole OT
withheld); the scores below are on withheld OT material (Ruth, Jonah,
Genesis 1) the model never saw in the target language. Target token
ilo_Latn (existing init — the token-init comparison in the experiment
showed the init method makes no measurable difference at an adequate
learning rate).
How to use
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained("DavidCBaines/ebible_m2o-nllb600m-control-ilo")
tokenizer = AutoTokenizer.from_pretrained("DavidCBaines/ebible_m2o-nllb600m-control-ilo")
tokenizer.src_lang = "mri_Latn" # any source language's FLORES code
batch = tokenizer(["<a mri verse>"], return_tensors="pt")
out = model.generate(
**batch,
forced_bos_token_id=tokenizer.convert_tokens_to_ids("ilo_Latn"),
num_beams=5, max_length=128,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
The forced_bos_token_id is also baked into generation_config.json, so
omitting it works too.
Evaluation (withheld OT books, never seen in ilo)
chrF3, sacreBLEU conventions. The source-copy floor is the chrF3 of the best source language's own text against the ilo reference — the score you would get by simply copying the closest relative. The model must beat it for the run to be publishable. Validation (250 NT verses): best chrF3 60.63.
| book | verses | best source | chrF3 | spBLEU | mean chrF3 over sources | source-copy floor |
|---|---|---|---|---|---|---|
| RUT | 85 | tgl | 62.43 | 36.12 | 55.15 | 29.06 |
| JON | 48 | ceb | 61.75 | 34.45 | 54.65 | 27.36 |
| GEN 1: | 31 | ceb | 61.11 | 34.51 | 58.44 | 23.42 |
Per-source detail:
- RUT: chrF3 by source — ceb 60.39, ind 51.35, mri 46.43, tgl 62.43
- JON: chrF3 by source — ceb 61.75, ind 47.97, mri 47.41, tgl 61.47
- GEN 1:: chrF3 by source — ceb 61.11, ind 57.45, mri 55.14, tgl 60.06
Training data and licensing
Fine-tuned on eBible-corpus translations. The base model (NLLB-200) is
CC-BY-NC-4.0, so this model is non-commercial regardless of the data
licences below; ShareAlike sources additionally propagate SA. Released under
cc-by-nc-sa-4.0.
| translation | language | FLORES code | licence |
|---|---|---|---|
iloulb | ilo (target) | — | by-sa |
mri2012 | mri | mri_Latn | by-nc |
indayt | ind | ind_Latn | by-nc |
cebulb | ceb | ceb_Latn | by-sa |
tglulb | tgl | tgl_Latn | by-sa |
Reproducibility
- Experiment:
m2o_control_ilo, initexisting, lr 0.0003, max 8000 steps, generation-based early stopping (chrF3 on a fixed 250-verse NT set, patience 3, min-delta 0.2). - Git commit:
423a6b234f489c111f6d44f0e45b53d621d8d720 - Code, configs and the full 15-run comparison: the project repository and the companion results dataset.
Acknowledgement
This series of experiments is inspired by the closed-text Bible translation experiment described by Sami Liedes in his blog. The repo was created with Claude Code using Opus 4.8 and Fable 5 with no code written by hand. There may well be errors present, but I trust Claude to do a better job than I can. Please let me know of any issues.