dormouse MT — English → Ukrainian (fine-tuned, v0.7)
Fine-tune of Helsinki-NLP/opus-mt-en-uk (76M params, runs on CPU). Closes the
loop for dormouse-ua: the English
answer from a cloud or local LLM comes back as fluent Ukrainian, without the
LLM generating a single Cyrillic token itself.
Training corpus (v0.7) — 282 945 pairs
| source | pairs | what it is |
|---|---|---|
inv | 158 521 | real Ukrainian chat (Telegram, Threads), inverted |
native | 97 912 | native English chat (hh-rlhf, oasst1), UA side MT |
cs | 24 702 | customer-support bitext, UA side MT |
surzhyk | 1 810 | synthetic surzhyk injected into real sentences |
The native block is what v0.6 was missing: earlier versions were trained
almost entirely on English that had itself been produced from Ukrainian
originals ("translationese"), which is a poor model of what an LLM actually
writes.
Metrics (v0.7, 845 held-out pairs, sacrebleu)
| slice | n | base BLEU | ft BLEU | base chrF | ft chrF |
|---|---|---|---|---|---|
| overall | 845 | 21.12 | 40.28 | 44.25 | 62.90 |
cs * | 250 | 27.49 | 68.15 | 49.35 | 82.49 |
native * | 250 | 26.93 | 49.43 | 49.75 | 68.97 |
inv | 250 | 9.70 | 19.97 | 33.97 | 45.26 |
surzhyk | 95 | 9.47 | 14.16 | 36.55 | 42.43 |
Read the starred slices with suspicion
* — on the cs and native slices the Ukrainian reference side was itself
produced by gemini-2.5-flash, not by a human. A 68 BLEU there means "the
fine-tune reproduces Gemini's Ukrainian well", which is what it was trained to
do. It is not evidence of human-level quality.
The honest slices are inv (real Ukrainian chat with human references) and
surzhyk. Read those first: 19.97 BLEU / 45.26 chrF, roughly double the
base model's 9.70 / 33.97.
That also corrects a tempting misreading. The overall en→uk number (40.28) is now higher than the uk→en sibling model (38.46), which looks like the reverse direction caught up — it did not. Strip the synthetic-reference slices and the picture is unchanged: 31.32 vs 19.97. Generating Ukrainian morphology is still the harder half of the round trip.
Known failure modes
- domain-shift nouns from outside chat:
brass→мідь,Necklaces→краватки,ring→обручка - ти/ви mixing inside one reply — the training chat data is inconsistent about register, and the model inherited that
Usage
from transformers import MarianMTModel, MarianTokenizer
m = MarianMTModel.from_pretrained("Dariachup/dormouse-mt-en-uk")
t = MarianTokenizer.from_pretrained("Dariachup/dormouse-mt-en-uk")
ids = t("So what's up with the order, three days silent", return_tensors="pt")
print(t.decode(m.generate(**ids, num_beams=4)[0], skip_special_tokens=True))
Translate one sentence at a time — MarianMT degrades noticeably on whole multi-line blocks.