japanese-reversed-gpt2-medium-unidic
This is a medium-sized Japanese reversed GPT-2 model using BERT-like tokenizer. Unlike most Language Models, this model generates sentences from right to left.
Not reversed version is published here.
How to use
The model depends on PyTorch, fugashi with unidic-lite, and Hugging Face Transformers.
pip install torch torchvision torchaudio
pip install fugashi[unidic-lite]
pip install transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
tokenizer = AutoTokenizer.from_pretrained('okazaki-lab/japanese-reversed-gpt2-medium-unidic')
model = AutoModelForCausalLM.from_pretrained('okazaki-lab/japanese-reversed-gpt2-medium-unidic')
text = 'ので、散歩に行きました。'
bos = tokenizer.convert_tokens_to_ids(['[BOS]']) # [32768]
input_ids = bos + tokenizer.encode(text)[1:-1][::-1] # [CLS] and [SEP] generated by BERT Tokenizer are removed then reversed
input_ids = torch.tensor(input_ids).unsqueeze(0)
output = model.generate(
input_ids,
do_sample=True,
max_new_tokens=30,
top_k=50,
top_p=0.95,
repetition_penalty=1.0,
num_return_sequences=1,
pad_token_id=0,
eos_token_id=32769,
)[0].flip(0)
print(tokenizer.decode(output))
Model architecture
Transformer-based Language Model
- Layers: 24
- Heads: 16
- Dimensions of hidden states: 1024
Training
We used a codebase provided by rinna Co., Ltd. for training.
The model was trained on Japanese CC-100 and Japanese Wikipedia (2022/01/31). We employed 8 A100 GPUs for 17 days. The perplexity on the validation set is 9.79.
Tokenization
Our tokenizer is based on the one provided by Tohoku NLP Group. The texts are tokenized by MeCab and then WordPiece.
The vocabulary size is 32771 (32768 original tokens + 2 special tokens + 1 unused token).
License
Creative Commons Attribution-ShareAlike 4.0
Copyright (c) 2021, Tohoku University
Copyright (c) 2023, Tokyo Institute of Technology