This is a pre-release checkpoint for a Nordic generative language model currently in training. This preliminary release is provided for HPLT (https://hplt-project.org/) deliverable 4.1 (“First language models trained”)(https://hplt-project.org/deliverables). Consult the HPLT website for further details. More documentation will be provided soon.
UPDATE: our Nordic model is now called Viking!
Viking 7B, 13B and 33B
NOTE: These are research checkpoint of a model for which training has not been completed. It is being provided in its current state for research and testing purposes. Care should be taken when using the outputs of the model. Once pretraining has completed we intend to release additional instruction-tuned and chat-tuned varieties.
Viking 7B, 13B and 13B are a 7B, 13B and 33B parameter decoder-only transformers pretrained on Finnish, English, Swedish, Danish, Norwegian, Icelandic and code. They are being trained on 2 trillion tokens (1.3 trillion as of this release).
Viking is a fully open source model and is made available under the Apache 2.0 License.
Viking was created in a collaboration between the TurkuNLP group of the University of Turku, SiloGen from Silo AI, and High Performance Language Technologies (HPLT). Training was conducted on the LUMI supercomputer, using compute resources generously provided by CSC - IT Center for Science, Finland.
This project is part of an ongoing effort to create open source large language models for non-English and especially low resource languages like Finnish. The mode is fluent in Finnish, English, the Scandinavian languages and capable of basic translation between them. It is also able to understand and generate code.
More info available at: