JaunBERT
Model Description
JaunBERT is a family of Basque encoder-only language models.
- Developed by: HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU)
- Funded by: IKERGAITU and ALIA projects (Basque and Spanish Governments)
- License: Apache 2.0
- Model Type: Encoder-only Transformer models (ModernBERT-style)
- Languages: Basque (Euskara)
Getting Started
You can either use this model directly as the example below, or fine-tune it to your task of interest.
>>> from transformers import pipeline
>>> pipe = pipeline("fill-mask", model='HiTZ/JaunBERT')
>>> pipe("Kaixo! Ni <mask> naiz!")
[{'score': 0.034423828125,
'token': 71333,
'token_str': 'Aitor',
'sequence': 'Kaixo! Ni Aitor naiz!'},
{'score': 0.032470703125,
'token': 119830,
'token_str': 'Asier',
'sequence': 'Kaixo! Ni Asier naiz!'},
{'score': 0.0208740234375,
'token': 109222,
'token_str': 'Unai',
'sequence': 'Kaixo! Ni Unai naiz!'},
{'score': 0.0196533203125,
'token': 164657,
'token_str': 'Nerea',
'sequence': 'Kaixo! Ni Nerea naiz!'},
{'score': 0.0196533203125,
'token': 40163,
'token_str': 'Ane',
'sequence': 'Kaixo! Ni Ane naiz!'}]
Training Data
The JaunBERT family was pre-trained on latxa-corpus-v2.
Each revision was further trained on a specific sub-corpus:
JaunBERT-legala: BOPV + BOG + BOTHA + ParlEusJaunBERT-periodistikoa: Euscrawl-v2JaunBERT-zientifikoa: Aldizkariak
Training objective is masked language modeling (MLM) on encoder-only architectures.
Evaluation
| AVG overall | |
|---|---|
| JaunBERT | 64.37 |
| JaunBERT-legala | 55.24 |
| JaunBERT-periodistikoa | 66.86 |
| JaunBERT-zientifikoa | 63.53 |
Acknowledgments
This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). Ekhi Azurmendi holds a PhD grant from the Basque Government (PRE_2024_1_0035, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042.