SIGMA-BERT

Table of Contents

1. Model Summary
2. Model Details
3. Usage
4. Training
5. Training Data
6. Tokenizer
7. Limitations
8. Citation

Model Summary

SIGMA-BERT is a Slovenian encoder language model based on the ModernBERT architecture. It was pretrained from scratch using the masked language modeling (MLM) objective on a diverse collection of Slovenian text, including web, news, parliamentary proceedings, scientific publications, and user-generated content. The model supports sequence lengths of up to 8192 tokens and is intended for downstream language understanding tasks such as classification, retrieval, semantic similarity, and token-level prediction.

As a pretrained encoder, SIGMA-BERT is intended to be fine-tuned for downstream Slovenian language understanding tasks such as:

• Text classification (e.g. topic, sentiment, genre)
• Named entity recognition and other token classification tasks
• Extractive question answering
• Natural language inference / semantic textual similarity
• Dense retrieval and semantic search, including over long documents
• As a backbone for rerankers in retrieval-augmented generation (RAG) pipelines

It is available in the following variants:

• SIGMA-BERT-short - 22 layers, 149 million parameters, 1,024 token context
• SIGMA-BERT-base - 22 layers, 149 million parameters, 8,192 token context
• SIGMA-BERT-large - 28 layers, 395 million parameters, 8,192 token context

Model Details

SigmaBERT-base is built on the ModernBERT architecture and pretrained on Slovenian text with a Masked Language Modeling (MLM) objective.

Training

SIGMA-BERT was pretrained in two stages.

Stage 1

The initial pretraining stage focused on learning general Slovenian language representations.

• Maximum sequence length: 1,024
• MLM masking probability: 30%
• Learning rate: 5e-4
• Optimizer: StableAdamW
• Weight decay: 0.1

Stage 2

The second stage extended the context window and continued pretraining on long-form, information-dense text.

• Maximum sequence length: 8,192
• MLM masking probability: 15%
• Learning rate: 5e-5
• Optimizer: StableAdamW
• Weight decay: 0.1
• Increased RoPE base frequency to support the longer context window

Both stages used a warmup–plateau–cooldown learning rate schedule.

Training Data

The training corpus contains approximately 13.88 billion tokens of Slovenian text.

Stage 1 used a mixture of publicly available Slovenian corpora:

• FineWeb2 — Slovenian subset (slv_Latn)
• Janes 1.0 (Wiki, Blog, Forum, News) — Slovenian text subcorpora
• OpenScience — Slovenian academic and scientific texts
• siParl 4.0 — Slovenian parliamentary proceedings corpus
• MaCoCu-sl 2.0 — Slovenian web corpus

Stage 2 continued pretraining using:

• OpenScience
• siParl 4.0

Tokenizer

SIGMA-BERT uses a custom SentencePiece BPE tokenizer trained from scratch with a vocabulary of 50,000 tokens. The tokenizer employs Metaspace pre-tokenization together with newline isolation and digit splitting to better preserve document structure and numeric information.

Usage

Load the model like any other transformers model:

from transformers import AutoTokenizer, AutoModel
 
tokenizer = AutoTokenizer.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-base")
model = AutoModel.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-base")

Masked language modeling

For MLM, load the model with AutoModelForMaskedLM and use the [MASK] token:

from transformers import AutoTokenizer, AutoModelForMaskedLM
 
model_id = "LHRS-UM-FERI/SIGMA-BERT-base"  
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
 
text = "Ljubljana je[MASK] mesto Slovenije."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
 
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
print("Predicted token:", tokenizer.decode(predicted_token_id))

⚠️ Note on [MASK] spacing: SIGMA-BERT's tokenizer uses a Metaspace pre-tokenizer, which encodes the space before a word as part of that word's token (e.g. " mesto" → ▁mesto). Because of this, if you write [MASK] in a text string, it should be placed directly against the preceding word, with no space ("je[MASK] mesto", not "je [MASK] mesto"). Adding a literal space before [MASK] produces an extra, out-of-distribution ▁ token that the model never saw during training:

"je [MASK]"  → ['▁je', '▁', '[MASK]', ...]   ✗ stray '▁' token
"je[MASK]"   → ['▁je', '[MASK]', ...]        ✓ matches training

Limitations

• SIGMA-BERT is an encoder-only model trained with masked language modeling and cannot generate text.
• The model is intended primarily for Slovenian.
• Like other pretrained language models, it may reflect biases present in the training data.

Citation

If you use SIGMA-BERT in your work, please cite:

@misc{Borovic2026,
  title = {SIGMA-BERT-base},
  author = {Mladen Borovič and Marija Jovanova},
  year = {2026},
  url = {https://huggingface.co/LHRS-UM-FERI/SIGMA-BERT-base},
  doi = { 10.57967/hf/9821 },
  publisher = { Hugging Face }
}