ScarletMT-Nano
A Japanese → English translation model (BART-based seq2seq), trained from scratch using the Hugging Face Trainer.
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("Yokii2/ScarletMT-Nano")
model = AutoModelForSeq2SeqLM.from_pretrained("Yokii2/ScarletMT-Nano")
inputs = tokenizer("ノバスコシア州ハリファックスにあるダルハウジー大学医学部教授でカナダ糖尿病協会の臨床・科学部門の責任者を務めるエフード・ウル博士は、この研究はまだ初期段階にあるとして注意を促しました。", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Evaluation
The model was evaluated on the Yokii2/jmdict-ja-en-bench benchmark (divided into short, medium, and long splits) using sacrebleu (BLEU, chrF2) and COMET22.
The table below displays the performance comparison with other models, ordered from highest to lowest overall performance:
| Model | Model Path | Benchmark | BLEU | chrF2 | COMET22 |
|---|---|---|---|---|---|
| Custom (v2) | Yokii2/quickmt-ja-en-v2 | short | 40.68 | 61.71 | 87.50 |
| Custom (v2) | Yokii2/quickmt-ja-en-v2 | medium | 39.95 | 65.36 | 89.86 |
| Custom (v2) | Yokii2/quickmt-ja-en-v2 | long | 37.52 | 64.64 | 88.38 |
| Base | quickmt/quickmt-ja-en | short | 33.45 | 53.83 | 84.35 |
| Base | quickmt/quickmt-ja-en | medium | 33.88 | 59.42 | 87.81 |
| Base | quickmt/quickmt-ja-en | long | 31.82 | 59.87 | 86.75 |
| ScarletMT-Nano | Yokii2/ScarletMT-Nano | short | 31.20 | 53.99 | 82.07 |
| ScarletMT-Nano | Yokii2/ScarletMT-Nano | medium | 30.15 | 58.87 | 86.50 |
| ScarletMT-Nano | Yokii2/ScarletMT-Nano | long | 29.03 | 58.80 | 85.33 |
| OPUS-MT | Helsinki-NLP/opus-mt-ja-en | short | 25.91 | 46.92 | 80.42 |
| OPUS-MT | Helsinki-NLP/opus-mt-ja-en | medium | 22.15 | 49.13 | 82.84 |
| OPUS-MT | Helsinki-NLP/opus-mt-ja-en | long | 20.26 | 48.65 | 81.15 |