Fugaku-LLM

Our Fugaku LLM model is a domestic model pre-trained from scratch using supercomputer Fugaku. This model is highly transparent and safety because it is trained from scratch with our own data. The training data consists mainly of Japanese data, and the model has excellent Japanese performance.

This models are developed by Fugaku-LLM.

Links to other models can be found in the index.

Fugaku-LLM Model Index

Model Details

• Developed by: Fugaku-LLM
• Model type: GPT-2
• Language(s): Japanese, English
• Library: DeepSpeedFugaku
• Tokenizer: llm-jp-tokenizer, code10k_en20k_ja30k of v2.2
• License: Fugaku-LLM Terms of Use

Model Performance

Instruction-tuned model

We evaluated our model by Japanese MT benchmark in the same way as Nejumi LLM Leaderboard Neo.
We only modified the following parts of the Fastchat code:

• Added "add_special_tokens=False" when calling tokenizer for input prompt
• Limit the number of tokens generated to less than 2048

How to use

Use the instruction-tuned model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "Fugaku-LLM/Fugaku-LLM-13B-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")
model.eval()

system_example = "以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。"
instruction_example = "スーパーコンピュータ「富岳」の名前の由来を教えてください。"

prompt = f"{system_example}\n\n### 指示:\n{instruction_example}\n\n### 応答:\n"

input_ids = tokenizer.encode(prompt,
                             add_special_tokens=False,
                             return_tensors="pt")
tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=128,
    do_sample=True,
    temperature=0.1,
    top_p=1.0,
    repetition_penalty=1.0,
    top_k=0
)
out = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(out)

Use the base model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "Fugaku-LLM/Fugaku-LLM-13B"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")
model.eval()

prompt = "スーパーコンピュータ「富岳」という名称は"

input_ids = tokenizer.encode(prompt,
                             add_special_tokens=False,
                             return_tensors="pt")
tokens = model.generate(
    input_ids.to(device=model.device),
    max_new_tokens=128,
    do_sample=True,
    temperature=0.1,
    top_p=1.0,
    repetition_penalty=1.0,
    top_k=0
)
out = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(out)

Training Datasets

Instruction Tuning

• oasst1
• databricks-dolly-15k
• gsm8k

License

Fugaku-LLM Terms of Use is available at LICENSE and LICENSE_ja files.

Risks and Limitations

The results of processing using Fugaku-LLM may contain falsehoods, biases, content that infringes on the rights of others, or content that does not meet the effectiveness or usefulness expected by Users.

Acknowledgements

This achievement is based on the Government‐Initiated Projects of Supercomputer Fugaku "Development of Distributed Training Method for Large Language Models on Fugaku."

Authors

• Tokyo Institute of Technology
• Tohoku University
• Fujitsu Limited
• RIKEN
• Nagoya University
• CyberAgent, Inc.
• Kotoba Technologies, Inc.