cat-minilm-l12 — MiniLM cross-encoder reranker (no score injection)
A MiniLM-L12 cross-encoder passage reranker trained on MS MARCO by knowledge distillation.
This is the matched baseline for Amdestya/bm25cat-minilm-l12,
built to reproduce the CECAT / CEBM25CAT comparison in
Askari et al., "Injecting the BM25 Score as Text Improves BERT-Based Re-rankers", ECIR 2023.
The two models share identical training data (the same 5,000,000 pairs), steps, optimizer, learning rate, batch size and sequence length. The only difference is the input construction: this one receives no injected BM25 score.
Usage
from sentence_transformers import CrossEncoder
model = CrossEncoder("Amdestya/cat-minilm-l12", max_length=256)
scores = model.predict([
["what is a cat", "A cat is a small domesticated carnivorous mammal."],
["what is a cat", "The 1998 federal budget deficit was revised upward."],
])
Input format
text_a = query
text_b = passage
# tokenises to: [CLS] query [SEP] passage [SEP]
Do not inject a BM25 score. This model was trained without one and has never seen a numeric token in
that position. For the injected variant use
Amdestya/bm25cat-minilm-l12, whose input format
differs — see its card.
Outputs are raw logits (identity activation), unbounded, higher = more relevant. They are comparable within a query, not across models.
Training
| base model | microsoft/MiniLM-L12-H384-uncased |
| objective | MSE against a BERTCAT teacher ensemble (Hofstätter et al.) |
| training pairs | 5,000,000 of 79,518,568 available |
| steps | 156,250 @ batch 32, 1 epoch |
| optimizer | Adam, lr 7e-6, 5,000 warmup steps |
| max length | 256 |
| hardware | 1 × RTX A5000 |
Note this is a knowledge-distillation setup, following the authors' released training notebook, rather than the cross-entropy objective described in the paper's §4.
Intended use
Built as a reference artifact for a dissertation study on reproducing IR papers with LLMs. Usable as a
general MS MARCO passage reranker, but the community
cross-encoder/ms-marco-MiniLM-L-12-v2
is trained more thoroughly and is the better choice for production use.