LLammas-base 🐑
Llama-2-7B with continued pre-training of 5B tokens of CulturaX (75% Estonian, 25% English documents).
This model is also instruction-tuned resulting in Llammas.
More details in our paper.
Citation
@misc{kuulmets2024teaching,
title={Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer},
author={Hele-Andra Kuulmets and Taido Purason and Agnes Luhtaru and Mark Fishel},
year={2024},
eprint={2404.04042},
archivePrefix={arXiv},
primaryClass={cs.CL}
}