This is the Llama-3b for Finnish model described in the paper Scaling Data-Constrained Language Models.

Llama-3b for Finnish

Pretrained Llama model on Finnish language using a causal language modeling (CLM) objective. Llama model was introduced in this paper and first released at this page.

There are two different sized Llama models, all pretrained from scratch for 131B tokens:

Note: this model is pretrained, not fine-tuned for instruction following or chat use. For instruction tuned models, you can check models listed in this collection: https://huggingface.co/collections/Finnish-NLP/instruction-tuned-models-65c08db79fa020161be2e942

Model description

Finnish Llama is a GPT-style model pretrained on a very large corpus of Finnish data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to predict the next word in sentences.

More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the predictions for the token i only uses the inputs from 1 to i but not the future tokens.

This way, the model learns an inner representation of the Finnish language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.

Intended uses & limitations

This model was only pretrained in a self-supervised way excluding any supervised training. You can use the this model for text generation or fine-tune it to a downstream task, like instruction following. Fine-tuning is recommended to get better results.

How to use

Here is an example of using this model in PyTorch with some generation arguments you can modify:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Finnish-NLP/llama-3b-finnish")
model = AutoModelForCausalLM.from_pretrained("Finnish-NLP/llama-3b-finnish")

inputs = tokenizer("Olipa kerran tekoälylaama", return_tensors="pt")
generated_ids = model.generate(inputs.input_ids, **{
      "temperature": 0.8,
      "penalty_alpha": 0.6,
      "top_k": 4,
      "do_sample": True,
      "repetition_penalty": 1.2,
      "min_length": 20,
      "max_length": 2048
    })
generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

Limitations and bias

The training data used for this model contains a lot of content from the internet, which is far from neutral. Therefore, the model can have biased predictions. This bias will also affect all fine-tuned versions of this model.

To reduce toxic content, training data was filtered with a toxicity classifier but it cannot truly eliminate all toxic text.

Training data

This Finnish Llama model was pretrained on the combination of 12 datasets:

• mc4_3.1.0_fi_cleaned, mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 version 3.1.0 dataset and further cleaned it with our own text data cleaning codes (check the dataset repo).
• oscar_2301_fi_cleaned, Oscar is a multilingual, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the Oscar version 23.01 dataset and further cleaned it with our own text data cleaning codes.
• wikipedia_20230501_fi_cleaned, we used the Finnish subset of the wikipedia (May 2023) dataset
• Reddit_fi_2006_2022, filtered and post-processed dataset of Finnish Reddit
• Yle Finnish News Archive 2011-2018
• Yle Finnish News Archive 2019-2020
• Finnish News Agency Archive (STT)
• The Suomi24 Sentences Corpus
• Project Lönnrot
• Finnish parliament speeches
• multilingual_cc_news, we used the Finnish subset of the multilingual CC-News dataset
• fi-news-corpus

Raw datasets were automatically cleaned to filter out bad quality and non-Finnish examples. Also, a perplexity score was calculated for all texts with a KenLM model which was trained with very clean Finnish texts only. This perplexity score can then be used to determine how "clean" Finnish language the text contains. To reduce toxic text, we used Finnish toxicity classifier TurkuNLP/bert-large-finnish-cased-toxicity released by TurkuNLP to classify all text examples. Classified toxicity label scores can then be used to determine how toxic the text is.

All datasets were concatenated and the whole dataset deduplicated. Top 95% perplexity score was used as a filtering threshold to filter out the worst quality 5% of texts. To reduce amount of toxic content, the dataset was filtered to include text examples having lower than 80% score for the toxicity labels "label_identity_attack", "label_insult", "label_threat" and "label_severe_toxicity".

Finally, 20,000 text examples from each of the Wikipedia, Yle, STT, Suomi24, and Reddit datasets were randomly selected for evaluation dataset in addition to the existing validation split of the mc4_3.1.0_fi_cleaned dataset.

The final training dataset had 19 billion words and the evaluation dataset had 25 million words. After tokenization, the training dataset had 28 billion tokens and the evaluation dataset had 38 million tokens. Training dataset is divided between different datasets as follows:

Training procedure

Preprocessing

Texts are tokenized using Byte Pair Encoding (BPE) based on sentencepiece with a vocabulary size of 64,256. Inputs are sequences of 2048 consecutive tokens. Texts are not lower cased so this model is case-sensitive: it makes a difference between finnish and Finnish.

Pretraining

The model was trained on TPUv4-32 VM, sponsored by the Google TPU Research Cloud. The model was trained for 131B tokens which is about 4.7 epochs of data, inspired by the findings of this paper. The optimizer used was a Lion, linear learning rate warmup for 2K steps to a peak learning rate of 3e-5, and then cosine decay to 3e-6.

Training was conducted with a slightly modified Jax/Flax based EasyLM framework, and inspired by the OpenLLaMA project.

Evaluation results

This model was evaluated using FIN-bench by TurkuNLP with zero-shot setting, but the evaluation script had some problems running succesfully, so the results reported below should perhaps be viewed with some caution.

llama-3b-finnish:

llama-7b-finnish:

| Task |Version| Metric |Value | |Stderr| |------------------------------------------------|------:|---------------------|-----:|---|-----:|
|bigbench_analogies | 0|multiple_choice_grade|0.2692|± |0.0391| |bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.2600|± |0.0441| |bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.3043|± |0.0981| |bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.2100|± |0.0409| |bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.1300|± |0.0338| |bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.0800|± |0.0273| |bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.2400|± |0.0429| |bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.0300|± |0.0171| |bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.1100|± |0.0314| |bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.1300|± |0.0338| |bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.0300|± |0.0171| |bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.1300|± |0.0338| |bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.0800|± |0.0273| |bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.0000|± |0.0000| |bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.4510|± |0.0704| |bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.7647|± |0.0600| |bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.5098|± |0.0707| |bigbench_emotions | 0|multiple_choice_grade|0.3688|± |0.0383| |bigbench_empirical_judgments | 0|multiple_choice_grade|0.3434|± |0.0480| |bigbench_general_knowledge | 0|multiple_choice_grade|0.1429|± |0.0421| |bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3793|± |0.0643| |bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3051|± |0.0605| |bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.4068|± |0.0645| |bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5814|± |0.0761| |bigbench_intent_recognition | 0|multiple_choice_grade|0.1850|± |0.0148| |bigbench_misconceptions | 0|multiple_choice_grade|0.5373|± |0.0432| |bigbench_paraphrase | 0|multiple_choice_grade|0.5000|± |0.0354| |bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.5333|± |0.0649| |bigbench_similarities_abstraction | 0|multiple_choice_grade|0.4474|± |0.0574|

Acknowledgements

This project would not have been possible without compute generously provided by Google through the TPU Research Cloud.

Team Members

• Aapo Tanskanen, Hugging Face profile, LinkedIn profile
• Rasmus Toivanen, Hugging Face profile, LinkedIn profile

Feel free to contact us for more details 🤗