opus-mt-tc-big-fr-zle — CTranslate2, int8
French → Russian (and the other East Slavic targets of the original model), converted to CTranslate2 with int8 weights. This is a format conversion of Helsinki-NLP/opus-mt-tc-big-fr-zle by the Language Technology Research Group at the University of Helsinki (OPUS-MT, Jörg Tiedemann and colleagues), released 2022-03-23. Nothing was retrained or fine-tuned. All credit for the model belongs to its authors.
It is the French-to-Russian model the NTranscript app downloads for its offline translation.
License
CC-BY 4.0, the license of the original model. The conversion is shared under the same terms.
Usage
Each source sentence starts with the target-language token >>rus<<
(>>ukr<< and >>bel<< also work), tokenized with source.spm; the output
is decoded with target.spm.
import ctranslate2, sentencepiece as spm
translator = ctranslate2.Translator(".", compute_type="int8")
src = spm.SentencePieceProcessor(model_file="source.spm")
tgt = spm.SentencePieceProcessor(model_file="target.spm")
tokens = [">>rus<<"] + src.encode("…", out_type=str) + ["</s>"]
result = translator.translate_batch([tokens], beam_size=2)
print(tgt.decode(result[0].hypotheses[0]))
Conversion
ct2-transformers-converter --model Helsinki-NLP/opus-mt-tc-big-fr-zle \
--output_dir opus-mt-tc-big-fr-zle-int8 --quantization int8 \
--copy_files source.spm target.spm
CTranslate2 4.8.2, transformers 5.17.0. int8 scores the same as float32 on the test we ran (WMT20 en-ru, first 500 sentences: BLEU 28.0 both), at half the size and about twice the speed on a CPU.
Files
| File | Bytes | SHA-256 |
|---|---|---|
config.json | 233 | 72901fbd8abd89fb5cf4a388f26fc681f5c4c58a1e1a88b30b879f107270e7ee |
model.bin | 242,439,853 | 742263b38109178d319fb67df179e54fe6ce9bed135f80940ee0ee78e440f7ad |
shared_vocabulary.json | 2,259,838 | 926e4fc129bdaa0d94d2c37371c7de1bab9547be728963447798bf61116794de |
source.spm | 824,258 | a552ab37e1013e599492a306a13f3bc6e163549d494ac162efc252f9d201a71f |
target.spm | 1,035,951 | 90e397a3ef99dc67d483405e86144399218afb25355aeb8caea483aef9b650b6 |
Original evaluation (fra-rus)
| language pair | test set | chr-F | BLEU | #sent | #words |
|---|---|---|---|---|---|
| fra-rus | tatoeba-test-v2021-08-07 | 0.66502 | 46.1 | 11490 | 70123 |
| fra-rus | flores101-devtest | 0.54106 | 25.8 | 1012 | 23295 |
| fra-rus | newstest2012 | 0.51254 | 23.1 | 3003 | 64790 |
| fra-rus | newstest2013 | 0.52342 | 24.8 | 3000 | 58560 |
Citation
@inproceedings{tiedemann-thottingal-2020-opus,
title = "{OPUS}-{MT} {--} Building open translation services for the World",
author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh},
booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
month = nov,
year = "2020",
address = "Lisboa, Portugal",
publisher = "European Association for Machine Translation",
url = "https://aclanthology.org/2020.eamt-1.61",
pages = "479--480",
}
@inproceedings{tiedemann-2020-tatoeba,
title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}",
author = {Tiedemann, J{\"o}rg},
booktitle = "Proceedings of the Fifth Conference on Machine Translation",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2020.wmt-1.139",
pages = "1174--1182",
}
Acknowledgements (from the original model card)
The original work is supported by the European Language Grid as pilot project 2866, by the FoTran project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 771113), and the MeMAD project, funded by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement No 780069, with computational resources provided by CSC — IT Center for Science, Finland.