Llama EstLLM 8B 0825 Base

Please note that this is a base text completion model that has not been instruction-tuned. It is intended for fine-tuning on downstream tasks rather than direct use for chat or instruction-following.

meta-llama/Llama-3.1-8B continuously pre-trained on approximately 35B tokens. Continued pre-training was performed for a single epoch on:

• Estonian National Corpus (8.6B tokens)
• Python-Edu (3.3B tokens)
• FineMath4-Plus (9.5B tokens)
• General Instruction-Augmented Corpora (7.4B tokens)
• Cosmopedia v2 (6.9B tokens)

Model Details

Model Description

• Developed by: TartuNLP and TalTechNLP research groups
• Funded by: Estonian Ministry of Education and Research, “Estonian Language Technology Program 2018-2027”
• Model type: Causal Language Model
• Language(s) (NLP): Estonian, English
• License: Llama 3.1 Community License Agreement
• Finetuned from model: meta-llama/Llama-3.1-8B

Evaluation

Logits-based

Estonian

English

Translation

Limitations

In addition to the limitations of the original Llama 3.1 8B model, this model has the following:

• Somewhat limited context size due to the continued training being done with the sequence length of 4096 tokens.

Citation

@misc{dorkin2026estllmenhancingestoniancapabilities,
      title={{EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training}}, 
      author={Aleksei Dorkin and Taido Purason and Emil Kalbaliyev and Hele-Andra Kuulmets and Marii Ojastu and Mark Fišel and Tanel Alumäe and Eleri Aedmaa and Krister Kruusmaa and Kairit Sirts},
      year={2026},
      eprint={2603.02041},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.02041}, 
}