marian-uk-verbalizer-ct2-int8
An int8 CTranslate2 export of
aloudreader/marian-uk-verbalizer,
a Ukrainian text-verbalization (TTS front-end) model. Same model, ~52 MB
instead of ~200 MB, for fast CPU or GPU inference.
Потрібно прочитати XX розділ до понеділка.
-> Потрібно прочитати двадцятий розділ до понеділка.
Full training/evaluation code and the router this model sits behind: GitHub repository.
Accuracy vs. the full-precision model
Compared against the PyTorch checkpoint on the same benchmarks:
| Set | Match rate (CT2 int8 vs. PyTorch fp32 output) |
|---|---|
| Frozen 3,597-row benchmark | 99.1–99.2% output-identical |
The int8 export's own scores on the per-domain evaluation sets are within 0.4 percentage points of the PyTorch checkpoint on every domain (usually identical), so it can be used as a drop-in low-latency replacement.
How to use
pip install ctranslate2 transformers sentencepiece huggingface_hub
import ctranslate2
from huggingface_hub import snapshot_download
from transformers import MarianTokenizer
path = snapshot_download("aloudreader/marian-uk-verbalizer-ct2-int8")
tokenizer = MarianTokenizer.from_pretrained(path)
translator = ctranslate2.Translator(path, device="cpu", compute_type="int8")
text = "Потрібно прочитати XX розділ до понеділка."
ids = tokenizer(text, truncation=True, max_length=384, add_special_tokens=True)["input_ids"]
tokens = tokenizer.convert_ids_to_tokens(ids)
# suppress_sequences is required: this export's decoder start token is "<s>"
# (id 0, not the usual pad token), which must never itself be generated.
result = translator.translate_batch([tokens], beam_size=1, suppress_sequences=[["<s>"]])
out_ids = tokenizer.convert_tokens_to_ids(result[0].hypotheses[0])
print(tokenizer.decode(out_ids, skip_special_tokens=True))
For GPU inference, pass device="cuda" to ctranslate2.Translator (requires
a CUDA build of CTranslate2 and libcublas).
Why the decoder-start handling matters
A naive Marian → CTranslate2 conversion assumes the decoder starts from a
zero embedding and uses the model's pad token as the decoder-start token.
This model's training used <s> (token id 0) as the decoder start instead;
converting it the naive way silently drops the first generated word from
every output. This export was produced with
scripts/convert_marian_ct2.py
in the GitHub repository, which sets start_from_zero_embedding=False and
the correct decoder-start token, and pairs it with
suppress_sequences=[["<s>"]] at inference time so <s> is never itself
generated as output.
License
MIT. See the GitHub repository.