KARAKURI LM 8x7B Chat v0.1
Model Details
Model Description
- Developed by: KARAKURI Inc.
- Model type: Mixture of Experts (MoE)
- Languages: Primarily English and Japanese
- License: Apache 2.0
- Finetuned from model: tokyotech-llm/Swallow-MX-8x7b-NVE-v0.1
- Contact: For questions and comments about the model, please email
karakuri-rd@karakuri.ai - Demo: https://lm.karakuri.cc/
Usage
Warning
The prompt format has been changed from KARAKURI LM 70B Chat v0.1. Please make sure to follow the correct format. Otherwise, the model will generate sub-optimal outputs.
Prompt Format
We use the following prompt template of multi-turn conversation in the Mistral format, which includes an encoded string of multiple attribute values.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("karakuri-ai/karakuri-lm-8x7b-chat-v0.1")
messages = [
{"role": "system", "content": "System prompt"},
{"role": "user", "content": "User prompt"},
{"role": "assistant", "content": "Model response"},
{"role": "user", "content": "User prompt"},
]
tokenizer.apply_chat_template(messages, tokenize=False)
# <s>[INST] <<SYS>>
# System prompt
# <</SYS>>
#
# User prompt [ATTR] helpfulness: 4 correctness: 4 coherence: 4 complexity: 4 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]Model response</s>[INST] User prompt [ATTR] helpfulness: 4 correctness: 4 coherence: 4 complexity: 4 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]
The prompt template contains nine attributes. The first five are derived from HelpSteer, while the remaining four are derived from OASST2. The values are represented by integers ranging from 0 to 4, with 0 being the lowest and 4 being the highest.
- helpfulness (default: 4): Overall helpfulness of the response to the prompt.
- correctness (default: 4): Inclusion of all pertinent facts without errors.
- coherence (default: 4): Consistency and clarity of expression.
- complexity (default: 4): Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).
- verbosity (default: 4): Amount of detail included in the response, relative to what is asked for in the prompt.
- quality (default: 4): Perceived goodness of response.
- toxicity (default: 0): Undesirable elements such as vulgar, harmful or potentially biased response.
- humor (default: 0): Sense of humor within response.
- creativity (default: 0): Willingness to generate non-conventional response.
If you want to change attribute values from the default values specified in the template, you can modify them to any values by adding the attribute values to the user messages:
messages = [
{"role": "user", "content": "User prompt", "helpfulness": 0, "complexity": 0},
]
tokenizer.apply_chat_template(messages, tokenize=False)
# <s>[INST] User prompt [ATTR] helpfulness: 0 correctness: 4 coherence: 4 complexity: 0 verbosity: 4 quality: 4 toxicity: 0 humor: 0 creativity: 0 [/ATTR] [/INST]
Run the model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"karakuri-ai/karakuri-lm-8x7b-chat-v0.1",
torch_dtype="auto",
device_map="auto",
)
messages = [
{
"role": "user",
"content": "週末に日帰りで東京に遊びに行こうと思っています。日帰りなので、短時間で回れるおすすめの観光プランを教えてください。",
},
]
input_ids = tokenizer.apply_chat_template(
messages,
return_tensors="pt",
).to(model.device)
outputs = model.generate(input_ids, max_new_tokens=512)
tokenizer.decode(outputs[0][input_ids.shape[-1]:])
Performance
| Model | # Active Params | Alignment | MT-Bench-jp |
|---|---|---|---|
| Qwen1.5 72B Chat | 72B | DPO | 8.19 |
| KARAKURI LM 8x7B Chat v0.1 | 13B | SteerLM | 7.54 |
| Command R+ | 104B | - | 7.31 |
| Mixtral 8x7B Instruct v0.1 | 13B | DPO | 7.24 |
| Llama 3 70B Instruct | 70B | RLHF | 7.13 |
| KARAKURI LM 70B Chat v0.1 | 70B | SteerLM | 6.43 |
| Llama 2 70B Chat | 70B | RLHF | 5.23 |
Training Details
Training Data
Training Infrastructure
- Hardware: The model was trained on 8 nodes of an Amazon EC2 trn1.32xlarge instance.
- Software: We use code based on neuronx-nemo-megatron.
Citation
@misc{karakuri_lm_8x7b_chat_v01,
author = { {KARAKURI} {I}nc. },
title = { {KARAKURI} {LM} 8x7{B} {C}hat v0.1 },
year = { 2024 },
url = { https://huggingface.co/karakuri-ai/karakuri-lm-8x7b-chat-v0.1 },
publisher = { Hugging Face },
journal = { Hugging Face repository }
}