KORMo-10B-base

🚀 Update News

• 2025-10-13: Official release of KORMo-10B-base (Be aware that it's not an SFT model!!).

💡 About KORMo

KORMo-10B is a 10.8B parameter fully open LLM capable of handling both Korean and English.
The model, training code, and training data are all fully open, allowing anyone to reproduce and extend them.

• Model Size: 10.8B parameters
• Languages: Korean / English
• Training Data: Synthetic data + public datasets (approximately 3T tokens)
• License: Apache 2.0

The First Fully Open-Source LLM from a Non-English Region

KORMo was created with a public-interest mission: to make world-class language models accessible to everyone.
Our goal is to empower anyone to build and advance their own large language models at a global standard.

Key Features:

1. A 10B-parameter Korean–English reasoning model trained entirely from scratch.
2. 100% open resources — including all training data, code, intermediate checkpoints, and tutorials — allowing anyone to reproduce and extend a near-SOTA model on their own.
3. 3 trillion tokens of training data released publicly, featuring never-before-shared, high-quality full-cycle Korean datasets (for pretraining, post-training, general, reasoning, and reinforcement learning).
4. A collaborative effort by eight master’s students at the KAIST Graduate School of Culture Technology (MLP Lab), documented in a 45-page research paper.

If you’ve ever used a Korean language model that performs well on benchmarks but feels strange in real use, or if fine-tuning only made it worse, you’re not alone.

KORMo solves these problems head-on.
By releasing every intermediate model and post-training dataset, we give users the freedom to build on the base model with their own data, customizing and fine-tuning it in any direction they want.

👉 "If you want a great Korean language model, now you can build it yourself. It even works with free Colab GPUs!" 🤗

🔗 Links

• 📖 Technical Report: 👉 Arxive
• 🤗 Hugging Face: 👉 Model Download
• 💻 GitHub Repository: 👉 Training and Inference Code
• 🔉 Tutorial: 👉 Instruction Tuning over google colab 👉 Youtube Tutorial

📈 Benchmark Performance

📊 Quantitative Evaluation

📝 Qualitative Evaluation (LLM-as-a-Judge)

📦 Installation

git clone https://github.com/MLP-Lab/KORMo-tutorial.git
cd KORMo-tutorial
bash setup/create_uv_venv.sh
source .venv_kormo/bin/activate

🚀 Inference Example

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "KORMo-Team/KORMo-10B-sft"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "user", "content": "What happens inside a black hole?"}
]

chat_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

inputs = tokenizer(chat_prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=1024,
    )

response = tokenizer.decode(output_ids[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print("Assistant:", response)

🧠 Enabling Thinking Mode

If you want to enable the thinking mode, simply set enable_thinking=True:

chat_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)

Contact

• KyungTae Lim, Professor at KAIST. ktlim@kaist.ac.kr

Acknowledgments

• This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (RS-2025-02653113, High-Performance Research AI Computing Infrastructure Support at the 2 PFLOPS Scale)

Citation

@misc{KORMo,
  author = {Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junghun Yuk, Changsu Choi, Dongjae Shin, Huije Lee, Hoyun Song, Alice Oh and KyungTae Lim},
  title = {KORMo: Korean Open Reasoning Model for Everyone},
  year = {2025},
  publisher = {GitHub},
  journal = {Technical Report},
  paperLink = {\url{https://arxiv.org/abs/2510.09426}},
 },
}