Sarashina2.1-1B
This repository provides large language models trained by SB Intuitions.
How to use
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2.1-1b", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2.1-1b")
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
set_seed(123)
text = generator(
"おはようございます、今日の天気は",
max_length=30,
do_sample=True,
pad_token_id=tokenizer.pad_token_id,
num_return_sequences=3,
)
for t in text:
print(t)
Model Description
We constructed this Sarashina2.1-1B model, which consists of 1 billion parameters, using a two-phase training process. First, we trained the model on 10 trillion tokens, including Japanese and English data extracted from web corpora. Then, we trained the model using 1 trillion tokens, predominantly consisting of Japanese data, to enhance its performance in Japanese. The following tables show the model's performance on Japanese and English tasks. We also show the performance of other public LLMs for reference.
Evaluation in Japanese tasks
| Model | Avg. | AIO | abc | JEMHopQA | NIILC | JComQA | JSQuAD |
|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B | 25.40 | 0.80 | 27.38 | 28.21 | 0.79 | 45.13 | 50.07 |
| Qwen2.5-1.5B | 39.61 | 7.00 | 38.14 | 27.35 | 11.81 | 79.18 | 74.18 |
| llm-jp-3-1.8B | 43.46 | 44.50 | 46.45 | 32.48 | 30.71 | 44.06 | 62.58 |
| llm-jp-3-3.7B | 54.24 | 54.10 | 49.63 | 36.75 | 49.61 | 58.36 | 77.01 |
| Sarashina2.1-1B (this model) | 58.31 | 54.70 | 58.44 | 41.88 | 48.82 | 64.70 | 81.34 |
Evaluation in English tasks
| Model | Avg. | PIQA | OpenBookQA | HellaSwag | Winogrande | ARC-easy | ARC-challenge |
|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B | 50.71 | 69.59 | 35.40 | 52.17 | 56.43 | 58.42 | 32.25 |
| Qwen2.5-1.5B | 60.84 | 76.17 | 40.40 | 67.83 | 63.85 | 72.01 | 44.80 |
| llm-jp-3-1.8B | 53.01 | 72.85 | 32.60 | 61.78 | 62.27 | 57.24 | 31.31 |
| llm-jp-3-3.7B | 56.70 | 74.92 | 36.60 | 67.75 | 62.90 | 61.91 | 36.09 |
| Sarashina2.1-1B (this model) | 56.01 | 74.10 | 37.20 | 63.16 | 61.01 | 63.64 | 36.95 |
Ethical Considerations and Limitations
Sarashina2.1 has not been tuned to follow an instruction yet. Therefore, sarashina2.1 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs. Before using sarashina2.1, we would like developers to tune models based on human preferences and safety considerations.