Model Card: OLAIR/ko-r1-7b-v2.0.3
This document describes the OLAIR/ko-r1-7b-v2.0.3 model, including its training data, intended use, performance benchmarks, limitations, and ethical considerations.
1. Overview
Model Name: OLAIR/ko-r1-7b-v2.0.3
Model Type: Large Language Model (LLM) for Korean language understanding and reasoning
Version: 2.0.3
This model is designed to provide Korean language capabilities with a focus on reasoning tasks. It is the second version in its series, building upon previous iterations with improvements in training data and fine-tuning methodologies.
2. Training Data
The model was trained on the dataset provided by OLAIR, specifically the Open-R1-Ko-SFT-v2.0 dataset. This dataset includes a curated collection of Korean language data, optimized for supervised fine-tuning (SFT) to enhance reasoning and natural language understanding capabilities in Korean.
3. Benchmark Performance
The model's performance has been evaluated using the HAE-RAE Reasoning Challenge (HRC), which measures reasoning abilities across various domains. Below are the benchmark results for several models, including OLAIR/ko-r1-7b-v2.0.3: We've noticed some errors in the previous code and updated it.
| Model | Chemistry | Math | Physics | Physics Word Puzzles | Puzzles | Average |
|---|---|---|---|---|---|---|
| o1-2024-12-17 | 57.14 | 78.18 | 77.78 | 80.00 | 84.62 | 75.54 |
| o3-mini-high | 57.14 | 81.82 | 77.78 | 70.00 | 69.23 | 71.19 |
| o3-mini-2025-01-31 | 50.00 | 80.00 | 70.37 | 50.00 | 76.92 | 65.46 |
| o1-mini-2024-09-12 | 42.86 | 56.36 | 70.37 | 60.00 | 15.38 | 48.99 |
| Deepseek-R1 | 50.00 | 54.55 | 62.96 | 70.00 | 7.69 | 49.04 |
| gpt-4o-2024-11-20 | 35.71 | 32.73 | 51.85 | 50.00 | 53.85 | 44.83 |
| Exaone-3.5-32B-Instruct | 21.43 | 30.91 | 25.93 | 50.00 | 38.46 | 33.35 |
| Qwen2.5-72B-Instruct | 35.71 | 30.91 | 51.85 | 20.00 | 23.08 | 32.31 |
| Ko-R1-7B-v2.0.3 | 7.14 | 61.82 | 40.74 | 40.00 | 0.00 | 29.94 |
| Ko-R1-7B-v1 | 7.14 | 63.64 | 37.04 | 40.00 | 0.00 | 29.56 |
| gpt-4o-mini-2024-07-18 | 21.43 | 29.09 | 37.04 | 50.00 | 0.00 | 27.51 |
| UNIVA-Bllossom_DeepSeek-llama3.1-Bllossom-8B | 28.57 | 16.36 | 33.33 | 10.00 | 15.38 | 20.73 |
| Model | Chemistry | Math | Physics | Physics Word Puzzles | Puzzles | Average |
|---|---|---|---|---|---|---|
| o1-2024-12-17 | 42.9 | 74.5 | 77.8 | 70.0 | 30.8 | 59.2 |
| o3-mini-high | 35.7 | 72.7 | 70.4 | 70.0 | 23.1 | 54.4 |
| o3-mini-2025-01-31 | 35.7 | 74.5 | 74.1 | 60.0 | 7.7 | 50.4 |
| o1-mini-2024-09-12 | 35.7 | 54.5 | 63.0 | 60.0 | 0.0 | 42.6 |
| Deepseek-R1 | 35.7 | 52.7 | 51.9 | 60.0 | 0.0 | 40.1 |
| gpt-4o-2024-11-20 | 28.6 | 21.8 | 37.0 | 50.0 | 0.0 | 27.5 |
| Ko-R1-7B-v2.0.3 | 7.1 | 56.4 | 29.6 | 40.0 | 0.0 | 26.6 |
| Qwen2.5-72B-Instruct | 35.7 | 29.1 | 37.0 | 30.0 | 0.0 | 26.4 |
| Ko-R1-7B-v1 | 0.0 | 60.0 | 22.2 | 40.0 | 0.0 | 24.4 |
| Exaone-3.5-32B-Instruct | 28.6 | 27.3 | 22.2 | 40.0 | 0.0 | 23.6 |
| gpt-4o-mini-2024-07-18 | 7.1 | 29.1 | 22.2 | 50.0 | 0.0 | 21.7 |
| UNIVA-Bllossom_DeepSeek-llama3.1-Bllossom-8B | 14.3 | 10.9 | 33.3 | 0.0 | 0.0 | 11.7 |
Note: The above table reflects performance across multiple reasoning domains. The metrics indicate that while OLAIR/ko-r1-7b-v2.0.3 shows competitive performance in certain areas (e.g., Math), there remain challenges, particularly in Chemistry and Physics-related tasks, compared to some higher-performing counterparts.
4. Limitations
- The model is still vulnerable to Korean-related inputs, leading to endless loops of thinking. We are working to fix it.
ETC
How to Cite
To be added
Contact
spthsrbwls123@yonsei.ac.kr