ModernBERT-tr-uncased-stsb-HD

Model Description

ModernBERT-tr-uncased-stsb-HD is a Turkish-specific hallucination detection model based on the ModernBERT architecture. This model is part of the Turk-LettuceDetect suite, specifically designed for detecting hallucinations in Turkish Retrieval-Augmented Generation (RAG) applications.

Model Details

• Model Type: Token-level binary classifier for hallucination detection
• Base Architecture: ModernBERT-base
• Language: Turkish (tr)
• Training Dataset: Machine-translated RAGTruth dataset (17,790 training instances)
• Context Length: Up to 8,192 tokens
• Model Size: ~135M parameters

Intended Use

Primary Use Cases

• Hallucination detection in Turkish RAG systems
• Token-level classification of supported vs. hallucinated content
• Real-time deployment in production Turkish NLP applications
• Quality assurance for Turkish text generation systems

Supported Tasks

• Question Answering (QA) hallucination detection
• Data-to-text generation verification
• Text summarization fact-checking

Performance

Overall Performance (F1-Score)

• Whole Dataset: 0.7821
• Question Answering: 0.7667
• Data-to-text Generation: 0.7733
• Summarization: 0.6341

Key Strengths

• Best overall performance across multiple Turkish RAG tasks
• Excellent balanced performance
• Strong performance in tasks (Summary and QA)
• Computational efficiency suitable for real-time applications

Training Details

Training Data

• Dataset: Machine-translated RAGTruth benchmark
• Size: 17,790 training instances, 2,700 test instances
• Tasks: Question answering (MS MARCO), data-to-text (Yelp), summarization (CNN/Daily Mail)
• Translation Model: Google Gemma-3-27b-it

Training Configuration

• Epochs: 6
• Learning Rate: 1e-5
• Batch Size: 4
• Hardware: NVIDIA A100 40GB GPU
• Training Time: ~2 hours
• Optimization: Cross-entropy loss with token masking

Data Processing

• Context and question tokens masked (label = -100)
• Answer tokens labeled as 0 (supported) or 1 (hallucinated)
• Binary classification task formulation

Technical Specifications

Architecture Features

• Positional Embeddings: Rotary Position Embeddings (RoPE)
• Attention Mechanism: Local-global attention
• Maximum Sequence Length: 8,192 tokens
• Classification Head: Binary token-level classifier

Input Format

Input: [CONTEXT] [QUESTION] [GENERATED_ANSWER]
Output: Token-level binary labels (0=supported, 1=hallucinated)

Limitations and Biases

Known Limitations

• Reduced effectiveness in summarization tasks compared to structured tasks
• Performance dependent on translation quality of training data
• Optimized specifically for Turkish; may not generalize to other languages
• Requires sufficient context for accurate hallucination detection

Potential Biases

• Translation artifacts from machine-translated training data
• Domain bias toward question answering and structured generation tasks
• Potential linguistic bias toward formal Turkish text patterns

Usage

Installation

pip install lettucedetect

Basic Usage

from lettucedetect.models.inference import HallucinationDetector

# Initialize the Turkish-specific hallucination detector
detector = HallucinationDetector(
    method="transformer", 
    model_path="newmindai/modernbert-tr-uncased-stsb-HD"
)

# Turkish context, question, and answer
context = "İstanbul Türkiye'nin en büyük şehridir. Şehir 15 milyonluk nüfusla Avrupa'nın en kalabalık şehridir."
question = "İstanbul'un nüfusu nedir? İstanbul Avrupa'nın en kalabalık şehri midir?"
answer = "İstanbul'un nüfusu yaklaşık 16 milyondur ve Avrupa'nın en kalabalık şehridir."

# Get span-level predictions (start/end indices, confidence scores)
predictions = detector.predict(
    context=context, 
    question=question, 
    answer=answer, 
    output_format="spans"
)

print("Tespit Edilen Hallusinasyonlar:", predictions)
# Örnek çıktı: 
# [{'start': 34, 'end': 57, 'confidence': 0.92, 'text': 'yaklaşık 16 milyondur'}]

Evaluation

Benchmark Results

Evaluated on machine-translated Turkish RAGTruth test set across three task types with consistent performance improvements over baseline multilingual approaches.

Example-level Results

Token-level Results

Citation

@inproceedings{turklettucedetect2025,
  title={Turk-LettuceDetect: A Hallucination Detection Models for Turkish RAG Applications},
  author={NewMind AI Team},
  booktitle={9th International Artificial Intelligence and Data Processing Symposium (IDAP'25)},
  year={2025},
  address={Malatya, Turkey}
}

Original LettuceDetect Framework

This model extends the LettuceDetect methodology:

@misc{Kovacs:2025,
      title={LettuceDetect: A Hallucination Detection Framework for RAG Applications}, 
      author={Ádám Kovács and Gábor Recski},
      year={2025},
      eprint={2502.17125},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.17125}, 
}

License

This model is released under an open-source license to support research and development in Turkish NLP applications.

Contact

For questions about this model or other Turkish hallucination detection models, please refer to the original paper or contact the authors.