Mandarin → Taiwanese Hokkien (opus-mt finetune)
Translates Mandarin into Taiwanese Hokkien. Finetuned by
Linguardia from Helsinki-NLP/opus-mt-zh-en on
Mandarin↔Taiwanese Hokkien sentence pairs from ChhoeTaigi dictionaries (github.com/ChhoeTaigi/ChhoeTaigiDatabase).
Built to generate course material for a language that no major machine translator supports.
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tok = AutoTokenizer.from_pretrained("linguardia/opus-mt-zh-nan")
model = AutoModelForSeq2SeqLM.from_pretrained("linguardia/opus-mt-zh-nan")
batch = tok([">>nan<< 我很早起床。"], return_tensors="pt")
out = model.generate(**batch, num_beams=4, max_length=64)
print(tok.batch_decode(out, skip_special_tokens=True))
The target language is selected by the >>nan<< prefix on the source
text. It was added as a new token rather than replacing one of the base
model's existing languages, so every language the base already handled still
works.
Training
| base | Helsinki-NLP/opus-mt-zh-en |
| training pairs | 106,678 |
| held out | 204 dev, 0 gold |
| epochs | 3.0 |
| batch size | 32 |
| optimiser | Adafactor, lr 5e-5, 500 warmup steps |
| max sequence | 48 tokens (99.9% of source, 99.6% of target) |
| hardware | Apple M4, MPS |
Evaluation
Held-out dev set (204 pairs never seen in training):
| metric | score |
|---|---|
| BLEU | 19.3 |
| chrF | 49.6 |
Training data and attribution
Sentence pairs come from ChhoeTaigi dictionaries (github.com/ChhoeTaigi/ChhoeTaigiDatabase). This model is a derivative of that work: if you use it, credit the corpus and its contributors under its own terms, recorded here as:
ChhoeTaigi dictionaries (github.com/ChhoeTaigi/ChhoeTaigiDatabase), CC BY-SA 4.0 and CC0 1.0; TaihoaSoanntengTuichiautian (BY-SA) + iTaigiHoataiTuichiautian (CC0) + TaioanPehoeKichhooGiku (BY-SA), Mandarin source side, normalised to Tai-lo
Pairs were deduplicated on exact (Mandarin, Taiwanese Hokkien) text. No machine translation was used to create the training data at any point.
Licence chain
| layer | licence |
|---|---|
base model Helsinki-NLP/opus-mt-zh-en | Apache-2.0 |
| training data | CC BY-SA 4.0 and CC0 1.0 |
| these weights | Apache-2.0, with the attribution above |
Limitations
- Trained on short everyday sentences. Expect it to be weakest on long, technical or literary text.
- Neural MT translates each sentence independently, so a term may be rendered differently across a corpus. Check consistency if that matters to you.
- Output should be reviewed by a speaker before being taught to learners.