Turkuaz-Embeddings

Overview

Turkuaz-Embeddings is a novel Turkish embedding model specifically optimized for information retrieval tasks. While multilingual embedding models aim to cover a wide range of languages, they often struggle with low-resource languages like Turkish. Turkuaz-Embeddings addresses these limitations, achieving significant improvements in retrieval performance across multiple Turkish benchmarks.

With the rise of Retrieval-Augmented Generation (RAG) systems, the need for reliable, language-specific embeddings has become critical. Turkuaz-Embeddings enhances retrieval reliability and accuracy, making it a valuable component for RAG pipelines and other downstream semantic search tasks involving Turkish content.

Highlights

• Outperforms widely used multilingual models with certain Turkish benchmarks, and Turkish embedding models by up to 20% and on average 9% on Turkish retrieval benchmarks.
• Achieves up to 35% and on average 20% improvement compared to its baseline architecture XLM-RoBERTa Large.
• Demonstrates robust zero-shot retrieval capabilities, indicating strong generalization.
• Specifically designed for Turkish, addressing the unique challenges of an agglutinative language.
• Embedding Size: 1024

Model Training Process

Turkuaz-Embeddings was developed based on the XLM-RoBERTa architecture and fine-tuned using the MSMARCO-TR dataset. The training pipeline involved multiple innovative steps:

• Training Data:

  • Document pairs with reranker-generated scores

• Reranker for Scoring:

  • Jina Reranker v2, a cross-encoder model, was used to generate more nuanced relevance scores for document pairs.

• Loss Function:

  • Trained with CoSENT (Cosine Sentence) Loss from the Sentence Transformers library, optimizing cosine similarity between sentence embeddings.

• Evaluation During Training:

  • Used the Embedding Similarity Evaluator from Sentence Transformers to assess embedding quality.

• Hardware and Training Setup:

  • Trained on 2 million document pairs.
  • Parallel distributed training over 4x Nvidia A100 80GB GPUs.
  • Learning rate: 5e-5 (initial phase), 5e-6 (later phases).
  • Warm-up ratios: 0.1 initially, 0.01 for subsequent phases.

Evaluation Results

📊 Leveraged Models

📊 Leveraged Datasets

📊 Retrieval Performance Comparison with Turkish Models

📊 Table 7: Retrieval Performance Comparison with Multilingual Models

Intended Use

• Semantic Search
• Retrieval-Augmented Generation (RAG) Systems
• Zero-shot Retrieval
• Turkish NLP Research and Applications

Sample Usage

from sentence_transformers import SentenceTransformer

# Load the model
device = "cuda"  # or "cpu"
model = SentenceTransformer("eneSadi/turkuaz-embeddings", device=device)

# Encode sentences
sentences = ["Ben bir ceviz ağacıyım Gülhane parkında.", "Herkes gömlek giyerken, Ahmet ceket giyerdi."]
embeddings = model.encode(sentences)

print(embeddings.shape)  # (2, 1024)

Keywords

Embedding Models, Information Retrieval, Semantic Search, Retrieval-Augmented Generation (RAG), Turkish NLP

Citation

If you use Turkuaz-Embeddings in your work, please consider citing it appropriately (citation format to be provided later).

License

Apache 2.0

Acknowledgements

This work leveraged the MSMARCO-TR dataset and builds upon the Sentence Transformers library and Jina Reranker v2 model.