CrossGemma Cross-Encoder: Semantic Similarity (STS)

Cross encoders are high performing encoder models that compare two texts and output a 0-1 score. I've found the cross-encoders/roberta-large-stsb model to be very useful in creating evaluators for LLM outputs. They're simple to use, fast and very accurate.

I trained this one using google's gemma encoder model google/embeddinggemma-300m.

Features

• High performing: Achieves Pearson: 0.9175 and Spearman: 0.9135 on the STS-Benchmark test set.
• Efficient architecture: Based on the Gemma-encoder design (300M parameters), offering very fast inference speeds.
• Extended context length: Processes sequences up to 2048 tokens, good for LLM output evals.
• Diversified training: Pretrained on dleemiller/wiki-sim and fine-tuned on sentence-transformers/stsb.

Performance

Usage

To use EttinX for semantic similarity tasks, you can load the model with the Hugging Face sentence-transformers library:

from sentence_transformers import CrossEncoder

# Load CrossEncoder model
model = CrossEncoder("dleemiller/CrossGemma-sts-300m")

# Predict similarity scores for sentence pairs
sentence_pairs = [
    ("It's a wonderful day outside.", "It's so sunny today!"),
    ("It's a wonderful day outside.", "He drove to work earlier."),
]
scores = model.predict(sentence_pairs)

print(scores)  # Outputs: array([0.9184, 0.0123], dtype=float32)

Output

The model returns similarity scores in the range [0, 1], where higher scores indicate stronger semantic similarity.

Training Details

Pretraining

The model was pretrained on the pair-score-sampled subset of the dleemiller/wiki-sim dataset. This dataset provides diverse sentence pairs with semantic similarity scores, helping the model build a robust understanding of relationships between sentences.

• Classifier Dropout: a somewhat large classifier dropout of 0.3, to reduce overreliance on teacher scores.
• Objective: STS-B scores from dleemiller/ModernCE-large-sts.

Fine-Tuning

Fine-tuning was performed on the sentence-transformers/stsb dataset.

Validation Results

The model achieved the following test set performance after fine-tuning:

• Pearson Correlation: 0.9175
• Spearman Correlation: 0.9135

Model Card

• Architecture: embeddinggemma-300m
• Tokenizer: Custom tokenizer trained with modern techniques for long-context handling.
• Pretraining Data: dleemiller/wiki-sim (pair-score-sampled)
• Fine-Tuning Data: sentence-transformers/stsb

Thank You

Thanks to the Google Deep Mind team for providing the encodergemma model, and the Sentence Transformers team for their leadership in transformer encoder models.

License

This model is licensed under the Apache 2.0.