bm25cat-minilm-l12 — MiniLM cross-encoder with the BM25 score injected as text
A MiniLM-L12 cross-encoder passage reranker that takes the normalised BM25 score as a text token in its input, reproducing the CEBM25CAT method of Askari et al., "Injecting the BM25 Score as Text Improves BERT-Based Re-rankers", ECIR 2023.
Its matched baseline — same data, same schedule, no injection — is
Amdestya/cat-minilm-l12.
Input format — read before using
text_a = f"{bm25_score} [SEP] {query}" # score BEFORE the query
text_b = passage
# tokenises to: [CLS] score [SEP] query [SEP] passage [SEP]
⚠️ This ordering differs from the paper's Eq. 3, which places the score between query and passage. This model follows the authors' released training code, which puts it before the query. A cross-encoder fed a layout it never saw in training returns confident nonsense rather than an error, so the ordering matters.
The score must be normalised the same way
bm25_score = int(((raw_bm25 - 0) / (50 - 0)) * 100) # Min-Max, GLOBAL setting, then x100 as an integer
Min-Max normalisation in the global setting — fixed constants min=0, max=50, not per-query
statistics — then scaled by 100 and truncated to an integer, per the paper's §3.3. Raw or per-query
normalised scores put the feature on a distribution the model has never seen.
Training scores came from pyserini / Lucene BM25Similarity(k1=0.82, b=0.68) (the Anserini-tuned MS
MARCO settings). Other BM25 implementations — Terrier's defaults are k1=1.2, b=0.75 — produce a
different score distribution, which shifts the injected integers.
Usage
from sentence_transformers import CrossEncoder
model = CrossEncoder("Amdestya/bm25cat-minilm-l12", max_length=256)
def norm(raw):
return int((raw / 50) * 100)
query = "what is a cat"
passage = "A cat is a small domesticated carnivorous mammal."
raw_bm25 = 36.5
score = model.predict([[f"{norm(raw_bm25)} [SEP] {query}", passage]])
Verify the tokenisation rather than assuming the literal [SEP] resolves as intended:
print(model.tokenizer.convert_ids_to_tokens(
model.tokenizer(f"73 [SEP] {query}", passage)["input_ids"]))
# ['[CLS]', '73', '[SEP]', 'what', 'is', 'a', 'cat', '[SEP]', 'a', 'cat', ...]
Three [SEP] tokens, with the score as its own token after [CLS]. Outputs are raw logits (identity
activation), unbounded, higher = more relevant.
Training
| base model | microsoft/MiniLM-L12-H384-uncased |
| objective | MSE against a BERTCAT teacher ensemble (Hofstätter et al.) |
| training pairs | 5,000,000 of 79,518,568 available |
| steps | 156,250 @ batch 32, 1 epoch |
| optimizer | Adam, lr 7e-6, 5,000 warmup steps |
| max length | 256 |
| hardware | 1 × RTX A5000 |
This is a knowledge-distillation setup, following the authors' released notebook, rather than the cross-entropy objective described in the paper's §4. Worth noting when interpreting results: the teacher ensemble's scores are computed without any BM25 input, so the training objective does not directly reward the model for using the injected feature.
Intended use
Built as a reference artifact for a dissertation study on reproducing IR papers with LLMs.