EttinX Cross-Encoder: Natural Language Inference (NLI)
This cross encoder performs sequence classification for contradiction/neutral/entailment labels. This has drop-in compatibility with comparable sentence transformers cross encoders.
To train this model, I added teacher logits to the all-nli dataset dleemiller/all-nli-distill from the
dleemiller/ModernCE-large-nli model. This significantly improves performance above standard training.
This 32m architecture is based on ModernBERT and is an excellent candidate for lightweight CPU inference.
Features
- High performing: Achieves 83.80% and 88.20% (Micro F1) on MNLI mismatched and SNLI test.
- Efficient architecture: Based on the Ettin-32m encoder design (32M parameters), offering faster inference speeds.
- Extended context length: Processes sequences up to 8192 tokens, great for LLM output evals.
Performance
| Model | MNLI Mismatched | SNLI Test | Context Length | # Parameters |
|---|---|---|---|---|
| dleemiller/ModernCE-large-nli | 0.9202 | 0.9110 | 8192 | 395M |
| dleemiller/ModernCE-base-nli | 0.9034 | 0.9025 | 8192 | 149M |
| cross-encoder/nli-deberta-v3-large | 0.9049 | 0.9220 | 512 | 435M |
| cross-encoder/nli-deberta-v3-base | 0.9004 | 0.9234 | 512 | 184M |
| cross-encoder/nli-distilroberta-base | 0.8398 | 0.8838 | 512 | 82M |
| dleemiller/EttinX-nli-xs | 0.8380 | 0.8820 | 8192 | 32M |
| dleemiller/EttinX-nli-xxs | 0.8047 | 0.8695 | 8192 | 17M |
Usage
To use EttinX for NLI tasks, you can load the model with the Hugging Face sentence-transformers library:
from sentence_transformers import CrossEncoder
# Load EttinX model
model = CrossEncoder("dleemiller/EttinX-nli-xs")
scores = model.predict([
('A man is eating pizza', 'A man eats something'),
('A black race car starts up in front of a crowd of people.', 'A man is driving down a lonely road.')
])
# Convert scores to labels
label_mapping = ['contradiction', 'entailment', 'neutral']
labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)]
# ['entailment', 'contradiction']
Training Details
Pretraining
We initialize the `` weights.
Details:
- Batch size: 512
- Learning rate: 1e-4
- Attention Dropout: attention dropout 0.1
Fine-Tuning
Fine-tuning was performed on the dleemiller/all-nli-distill dataset.
Validation Results
The model achieved the following test set micro f1 performance after fine-tuning:
- MNLI Unmatched: 0.8380
- SNLI: 0.8820
Model Card
- Architecture: Ettin-encoder-32m
- Fine-Tuning Data:
dleemiller/all-nli-distill
Thank You
Thanks to the Johns Hopkins team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models.
Citation
If you use this model in your research, please cite:
@misc{moderncenli2025,
author = {Miller, D. Lee},
title = {EttinX NLI: An NLI cross encoder model},
year = {2025},
publisher = {Hugging Face Hub},
url = {https://huggingface.co/dleemiller/EttinX-nli-xxs},
}
License
This model is licensed under the MIT License.