ettin-encoder-68m-contrastive-2.5m
Weakly-supervised contrastive pre-training of jhu-clsp/ettin-encoder-68m on ~2.5 million (query, passage) pairs drawn from three open-domain datasets.
This is Stage 1 of a two-stage training pipeline. The model produces 512-dimensional sentence embeddings via mean pooling and is a strong starting point for task-specific retrieval fine-tuning.
Model details
| Property | Value |
|---|---|
| Base model | jhu-clsp/ettin-encoder-68m (ModernBERT encoder) |
| Architecture | ModernBERT — 68M parameters |
| Embedding dimension | 512 |
| Pooling | Mean pooling over last-layer token embeddings |
| Max sequence length | 256 tokens |
| Weight dtype | float32 |
| Training precision | bf16 (autocast) |
Training
Datasets
Training pairs come from three sources, interleaved with fixed probabilities (seed 42):
| Dataset | Pairs | Probability | Pair type |
|---|---|---|---|
| GooAQ | ~3.0M | 58% | question → long-form answer |
Wikipedia Sections (pair config) | ~2.0M | 38% | section title → section body |
| Natural Questions | ~100K | 4% | question → Wikipedia passage |
Interleaving stops when NQ exhausts, giving 2,504,448 total training pairs (~9,783 gradient steps at batch size 256).
Objective
CachedMultipleNegativesRankingLoss with in-batch negatives. Mini-batch size of 32 is used for embedding (cached), while the contrastive loss is computed over the full effective batch of 256 — enabling large-batch training on 8 GB VRAM.
Hyperparameters
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning rate | 2e-5 |
| LR schedule | Linear with 5% warmup |
| Effective batch size | 256 |
| Mini-batch size (embedding) | 32 |
| Epochs | 1 |
| Max sequence length | 256 |
Hardware
Single NVIDIA RTX 4060 Laptop (8 GB VRAM).
Evaluation
Evaluated on NanoBEIR (13 subsets, NDCG@10) after Stage 1 training:
| Metric | Score |
|---|---|
| NanoBEIR NDCG@10 | 0.3572 |
For comparison, the base model (jhu-clsp/ettin-encoder-68m) without any contrastive training achieves NanoBEIR NDCG@10 of ~0.10.
After an additional Stage 2 hard-negative fine-tuning on MS MARCO, the combined pipeline reaches NanoBEIR NDCG@10 of 0.5145 — see capemox/ettin-encoder-68m-msmarco-combined (coming soon).
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("capemox/ettin-encoder-68m-contrastive-2.5m")
queries = ["What is the capital of France?"]
passages = ["Paris is the capital and most populous city of France."]
q_emb = model.encode(queries, normalize_embeddings=True)
p_emb = model.encode(passages, normalize_embeddings=True)
scores = q_emb @ p_emb.T
print(scores)
Citation
If you use this model, please cite the original Ettin paper:
@misc{weller2025seqvsseq,
title = {Seq vs Seq: An Open Suite of Paired Encoders and Decoders},
author = {Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
year = {2025},
eprint = {2507.11412},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2507.11412}
}