NorOLMo

NorOLMo 1.0 is a base (not instruction-tuned) large language model, continually pre-trained on Norwegian and Scandinavian data starting from the English OLMo2-13B model.

It is the only modern fully open large language model specifically adapted for Norwegian and Sámi. At the same time, its performance is comparable to other multilingual open models (see the Evaluation section below). "Fully open" here means that the entirety of the trainining data and training pipeline is transparent and reproducible, not only that the model weights are available.

The model was trained for 33 000 steps on around 275 billion tokens. Maximum sequence length was set to 4 096 in the beginning of training, and then was extended to 16 384 starting from the checkpoint step_31000. Intermediate checkpoints are published here as branches. The main branch contains the model's weights after step 33 000 (stage 3).

In a few months, we hope to follow with post-trained versions of NorOLMo.

Evaluation

Below is a comparison of fully-open models supporting Norwegian. The figure shows the aggregate score across all 35 NorEval 1.1 tasks (5 categories, category average). Scores are first averaged within each task category, then averaged across categories. This gives equal weight to each category regardless of how many tasks it contains. Each task score is normalized to a 0–100 scale where 0 = random baseline performance and 100 = perfect score, then averaged across tasks. This accounts for different chance levels across tasks (e.g. 25% for 4-choice QA vs. 50% for binary classification).

More detailed evaluation is evailable in our interactive NorEval dashboard: https://ltgoslo.github.io/llm-dashboard.

Furthermore, an interactive per-checkpoint evaluation with additional ablation studies is available here.

Data Details

Stage 1 (24 000 steps -- 200B tokens)

Data ("pretraining data")

• HPLTv3: Bokmål, Nynorsk, Faroese, Icelandic, Danish, Swedish
• FinePDFs: Bokmål, Nynorsk, Faroese, Icelandic, Danish, Swedish
• OLMo-Mix
• Northern Sámi: (Glot500, Northern Sámi Web Corpus , SIKOR North Saami corpus)

[!NOTE] The number of documents represents the total unique number of documents, not the documents used during training.

[!NOTE] We only took a portion of OLMo-Mix as our unique data.

Stage 2 (6 000 steps -- 50B tokens) and Stage 3 (3 000 steps -- 25B tokens)

Data ("midtraining data")

• HPLTv3 (filtered): Bokmål, Nynorsk, Icelandic, Danish, Swedish
• FinePDFs-Edu: Bokmål, Nynorsk, Icelandic, Danish, Swedish, English
• FinePDFs: Faroese
• Northern Sámi: (Glot500, Northern Sámi Web Corpus , SIKOR North Saami corpus)
• Stack-Edu
• MegaMath Web-Pro
• FineMath 4+
• InfiWebMath 4+

Data Splits

Training details

Acknowledgements

Training was conducted as a part of the HPLT project.

This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]