Please note that access is limited to students, companies, and organizations from Nordic countries. Kindly provide your work email or student email to access the models. Thank you for your understanding.
Model Details
NorwAI-Mixtral-8x7B is continue-pretrained on Mixtral-8x7B-v0.1 using 51B tokens. It belongs to NowAI LLM family developed by NowAI research center at Norwegian University of Science and Technology (NTNU) in collaboration with Schibsted, NRK, VG and the National Library of Norway.
NorwAI LLM includes a collection of pretrained, continue pretrained and instruction tunned generative text models in 7B and 45B sizes with different archtectures. All pretrained and continue pretrained models are on the same dataset and with the same tokenizer. The instruction tuned models are optimized using high-quality Norwegian instructions collected from Norwegian native speakers. The development of these modesl is dedicated to promoting and developing the research and applications of Norwegian language models.
- Developed by: NowAI at NTNU, Schibsted and VG
- Model type: Generative text model
- Language(s) (NLP): Norwegian
- Tokenizer: We expanded the Norwegian vocabulary by merging the Llama 2 tokenizer with the vocabulary from our own trained Norwegian tokenizer. The extended vocabulary size is 64000.
- Models release date: May 15, 2024 but are being continuously updated.
NowAI LLM family is based on auto-regressive language model architecture.
| model_name | #parameter | training scheme | context length | base model |
|---|---|---|---|---|
| NorwAI-Mistral-7B | 7B | continue-pretrain | 32k | Mistral-7B-v0.1 |
| NorwAI-Mistral-7B-pretrain | 7B | pretrain from scratch | 32k | Mistral-7B-v0.1 |
| NorwAI-Llama2-7B | 7B | continue-pretrain | 4096 | Llama2 |
| NorwAI-Mixtral-8x7B | 45B | continue-pretrain | 32k | Mixtral-8x7B-v0.1 |
| NorwAI-Mistral-7B-instruct | 7B | instruction tuning | 32k | NorwAI-Mistral-7B |
| NorwAI-Mixtral-8x7B-instruct | 45B | instruction tuning | 32k | NorwAI-Mixtral-8x7B |
Uses
NowAI LLM is intended for both commercial and research use in Nordic countries. To get access to the model, please carefully read the message and complete the required information.
Bias, Risks, and Limitations
The model may have potential risks common to large language models, such as hallucination, factual inconsistency, toxicity, and bias etc.
How to use
Let see the following example to load the model:
from transformers import AutoTokenizer, AutoModelForCausalLM
model_and_tokenizer_path = "NorwAI/NorwAI-Mixtral-8x7B"
access_token = "<your access token>"
# import tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_and_tokenizer_path, token=access_token)
model = AutoModelForCausalLM.from_pretrained(model_and_tokenizer_path, token=access_token, device_map='balanced')
# define your own prompt
prompt = """Hans Hyldbakk fra Surnadal var en"""
# generate response
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs,
min_new_tokens=50,
max_new_tokens=100,
do_sample=True,
temperature=0.3)
outputs = tokenizer.decode(outputs[0], skip_special_tokens=True)
Training Details
Training Data
All NorwAI LLM models were (continue-)pretrained on 51.15 Billion tokens, equivalent to 30.33 Billion words, sourced from public datasets and data shared by Schibsted, NRK, and VG partners under agreement. The publicly available datasets were preprocessed to filter out texts with copyright issues, and all datasets were preprocessed to remove sensitive information. Among all the pretraining data, the ratio of Norwegian to other languages is 3 to 2. Among the languages other than Norwegian, Swedish and Danish account for the majority, with a small amount of English and other languages.
Training Infrastructure
All models were pretrained and fine-tuned using the llm-foundary framework on the IDUN cluster in NTNU.
Model Card Contact
Please contact the following people if you have any questions regarding the models:
Lemei Zhang, lemei.zhang@ntnu.no
Peng Liu, peng.liu@ntnu.no
Citation Information
If you feel our work is helpful, please cite our papers:
@article{gulla2026norwai,
title={NorwAI's Large Language Models: Technical Report},
author={Gulla, Jon Atle and Liu, Peng and Zhang, Lemei},
journal={arXiv preprint arXiv:2601.03034},
year={2026}
}
@inproceedings{liu2024nlebench+,
title={NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian},
author={Liu, Peng and Zhang, Lemei and Farup, Terje and Lauvrak, Even and Ingvaldsen, Jon and Eide, Simen and Gulla, Jon Atle and Yang, Zhirong},
booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
pages={5543--5560},
year={2024}
}