PLaMo-100B
Model Description
PLaMo-100B is a 100B model pre-trained on English and Japanese open datasets, developed by Preferred Elements, Inc. PLaMo-100B is released under both Commercial and Non-Commercial Licenses. Please check the LICENSE for non-commercial use, both Japanese version and English version of the license are available. For commercial use, please contact us via this form （Japanese Only）.
NOTE: This model has NOT been instruction-tuned for chat dialog or other downstream tasks. We provide instruction-tuned version of PLaMo-100B model via our API and solution packages. Please check our official PLaMo website (Japanese only) for details.
Usage
Requirements
- numpy
- sentencepiece
- torch
- transformers
Use a pipeline as a high-level helper
import transformers
pipeline = transformers.pipeline("text-generation", model="pfnet/plamo-100b", trust_remote_code=True)
print(pipeline("The future of artificial intelligence technology is ", max_new_tokens=32))
Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("pfnet/plamo-100b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("pfnet/plamo-100b", trust_remote_code=True)
text = "これからの人工知能技術は"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_tokens = model.generate(
inputs=input_ids,
max_new_tokens=32,
do_sample=True,
top_k=50,
top_p=0.95,
temperature=1.0,
)[0]
generated_text = tokenizer.decode(generated_tokens)
print(generated_text)
Model Details
- Model size: 100B
- Trained tokens: 2T tokens (English: 1.3T tokens, Japanese: 0.7T tokens)
- Developed by: Preferred Elements, Inc
- Model type: Causal decoder-only
- Language(s): English, Japanese
- License: Commercial and Non-Commercial
Training Dataset
We trained PLaMo-100B in two phases, phase 1 with 1.5T tokens and phase 2 with 0.5T tokens. The percentage of datasets in each phase is shown in the following table.
| 1.5T (phase 1) | 0.5T (phase 2) | |
|---|---|---|
| RefinedWeb (English) | 42% | 17% |
| Other English Dataset | 28% | 33% |
| Proprietary CommonCrawl-JP | 18% | 46% |
| Other Japanese Dataset | 12% | 4% |
Tokenizer
PLaMo-100B uses sentencepiece tokenizer which is trained on a subset of the datasets for model pre-training.
Tech Blog
https://tech.preferred.jp/ja/blog/plamo-100b/
Bias, Risks, and Limitations
PLaMo-100B is a new technology that carries risks with use. Testing conducted to date has been in English and Japanese, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, PLaMo-100B’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of PLaMo-100B, developers should perform safety testing and tuning tailored to their specific applications of the model.
How to cite
@article{plamo100b,
author = {Preferred Elements, Inc. and Kenshin Abe and Kaizaburo Chubachi and Yasuhiro Fujita and Yuta Hirokawa and Kentaro Imajo and Toshiki Kataoka and Hiroyoshi Komatsu and Hiroaki Mikami and Tsuguo Mogami and Shogo Murai and Kosuke Nakago and Daisuke Nishino and Toru Ogawa and Daisuke Okanohara and Yoshihiko Ozaki and Shotaro Sano and Shuji Suzuki and Tianqi Xu and Toshihiko Yanase},
title = {PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency},
year = {2024},
url = {https://arxiv.org/abs/2410.07563},
journal = {arXiv}
}
Acknowledgement
This model is trained under the project, “Research and Development Project of the Enhanced Infrastructures for Post 5G Information and Communication System” (JPNP 20017), subsidized by the New Energy and Industrial Technology Development Organization (NEDO).