Japanese to English translator
Japanese to English translator model based on EncoderDecoderModel(bert-japanese+GPT2)
Usage
Demo
Please visit https://huggingface.co/spaces/sappho192/jesc-ja-en-translator-demo
Dependencies (PyPI)
- torch
- transformers
- fugashi
- unidic-lite
Inference
import transformers
import torch
encoder_model_name = "cl-tohoku/bert-base-japanese-v2"
decoder_model_name = "openai-community/gpt2"
src_tokenizer = transformers.BertJapaneseTokenizer.from_pretrained(encoder_model_name)
trg_tokenizer = transformers.PreTrainedTokenizerFast.from_pretrained(decoder_model_name)
model = transformers.EncoderDecoderModel.from_pretrained("sappho192/jesc-ja-en-translator")
def translate(text_src):
embeddings = src_tokenizer(text_src, return_attention_mask=False, return_token_type_ids=False, return_tensors='pt')
embeddings = {k: v for k, v in embeddings.items()}
output = model.generate(**embeddings, max_length=512)[0, 1:-1]
text_trg = trg_tokenizer.decode(output.cpu())
return text_trg
texts = [
"逃げろ!", # Should be "run!"
"初めまして.", # "nice to meet you."
"よろしくお願いします.", # "thank you."
"夜になりました", # "and then it got dark."
"ご飯を食べましょう." # "let's eat."
]
for text in texts:
print(translate(text))
print()
Dataset
The dataset used to train the model is JESC(Japanese-English Subtitle Corpus).
Its license is CC-BY-SA-4.0.
All data information can be accessed through following links:
- Dataset link: https://nlp.stanford.edu/projects/jesc/
- Paper link: https://arxiv.org/abs/1710.10639
- Github link: https://github.com/rpryzant/JESC
- Bibtex:
@ARTICLE{pryzant_jesc_2017,
author = {{Pryzant}, R. and {Chung}, Y. and {Jurafsky}, D. and {Britz}, D.},
title = "{JESC: Japanese-English Subtitle Corpus}",
journal = {ArXiv e-prints},
archivePrefix = "arXiv",
eprint = {1710.10639},
keywords = {Computer Science - Computation and Language},
year = 2017,
month = oct,
}