Ara-EuroBERT: Arabic-optimized Sentence Transformer

Ara-EuroBERT is a sentence-transformers model fine-tuned from EuroBERT/EuroBERT-210m specifically optimized for Semantic Arabic text embeddings.

This model maps sentences and paragraphs to a 768-dimensional dense vector space and Maximum Sequence Length: 8,192 tokens.

Paper:

You can find more information on the base model at https://huggingface.co/EuroBERT/EuroBERT-210m

Our fine-tuned model shows remarkable improvements over the base models, achieving a 73.5% relative improvement on STS17 and a 21.6% relative improvement on STS22.v2 compared to the base EuroBERT-210M.

Model Details

Model Description

• Model Type: Sentence Transformer with Matryoshka Embeddings
• Base model: EuroBERT/EuroBERT-210m
• Maximum Sequence Length: 8,192 tokens
• Output Dimensionality: Matryoshka embeddings with dimensions [768, 512, 256, 128, 64]
• Similarity Function: Cosine Similarity
• Languages: Optimized for Arabic
• License: Same as EuroBERT (MIT)

Matryoshka Embeddings

This model is trained with Matryoshka Representation Learning, allowing for flexible embedding dimensionality without retraining. You can use smaller dimensions (64, 128, 256, 512) for efficiency or the full 768 dimensions for maximum performance. The model maintains strong performance even at reduced dimensions:

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: EuroBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Use Cases

This model excels at various Arabic NLP tasks:

• Semantic textual similarity
• Semantic search and information retrieval
• Document clustering and classification
• Question answering
• Paraphrase detection
• Zero-shot classification

Usage

Installation

pip install -U sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("Omartificial-Intelligence-Space/AraEuroBert-210M")

# Encode Arabic sentences
sentences = [
    'التقدم العلمي في مجال الذكاء الاصطناعي يتسارع بشكل ملحوظ في السنوات الأخيرة',
    'تطوير نماذج لغوية متقدمة يساهم في تحسين فهم اللغة العربية آليًا',
    'أصبحت تقنيات معالجة اللغات الطبيعية جزءًا أساسيًا من التطبيقات الحديثة',
    'يعاني الشرق الأوسط من تحديات مناخية متزايدة تهدد الأمن المائي والغذائي',
    'تراث الأدب العربي غني بالقصائد والروايات التي تعكس تاريخ وثقافة المنطقة',
]
embeddings = model.encode(sentences)
print(embeddings.shape)  # [3, 768]

# Get the similarity scores
from sentence_transformers import util
similarities = util.cos_sim(embeddings, embeddings)
print(similarities)

Using Matryoshka Embeddings

# Get embeddings with different dimensions
embeddings_768 = model.encode(sentences)  # Default: full 768 dimensions
embeddings_256 = model.encode(sentences, truncate_dim=256)  # Use only 256 dimensions
embeddings_64 = model.encode(sentences, truncate_dim=64)  # Use only 64 dimensions

Training Method

• Loss Function: MatryoshkaLoss with MultipleNegativesRankingLoss
• Matryoshka Dimensions: [768, 512, 256, 128, 64]
• Batch Size: 32
• Epochs: 1 (with early stopping)
• Optimizer: AdamW
• Learning Rate: 5e-05 with linear scheduler and 10% warmup
• Hardware: Multiple NVIDIA GPUs with mixed precision (fp16)

Base Model: EuroBERT

EuroBERT is a new family of multilingual encoder models designed specifically for European and widely spoken global languages. It offers several advantages over traditional multilingual encoders:

• Extensive Multilingual Coverage: Trained on a 5 trillion-token dataset across 15 languages
• Advanced Architecture: Uses grouped query attention, rotary position embeddings, and RMS normalization
• Long Context Support: Natively processes up to 8,192 tokens
• Specialized Knowledge: Includes math and programming language data for improved reasoning

Limitations and Recommendations

• The model is primarily optimized for Arabic text and may not perform optimally on other languages
• Performance may vary for specialized domains not well-represented in the training data
• For short texts (<5 words), consider augmenting with context for better representations
• For extremely long documents, consider splitting into meaningful chunks before encoding

Citation

If you use this model in your research, please cite the following works:

@misc{boizard2025eurobertscalingmultilingualencoders,
      title={EuroBERT: Scaling Multilingual Encoders for European Languages}, 
      author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes and Ricardo Rei and Pierre Colombo},\
      year={2025},\
      eprint={2503.05500},\
      archivePrefix={arXiv},\
      primaryClass={cs.CL},\
      url={https://arxiv.org/abs/2503.05500}, 
}

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},\
    eprint={2205.13147},\
    archivePrefix={arXiv},\
    primaryClass={cs.LG}\
}