AtmicReRanker (MonoBERT cross-encoder)
LambdaRank-weighted pairwise fine-tune of castorini/monobert-large-msmarco on the combined
layer1+layer2 Ramana Maharshi Q&A reranking dataset (7,328 training queries / 1,280 validation
queries), max_length=512.
BREAKING CHANGE: this repo previously held a sentence-transformers-compatible bge-reranker-base
CrossEncoder checkpoint. It now holds a legacy PyGaggle-style MonoBERT checkpoint
(BertConfig+BertForSequenceClassification, num_labels=2), which is not loadable via
sentence_transformers.cross_encoder.CrossEncoder.
How to load
Loads exactly the same way as the base model it was fine-tuned from — the only difference is the repo id (and therefore the weights); architecture, tokenizer format, and scoring convention are identical, so you can swap between base and fine-tuned by changing one string:
from transformers import BertConfig, BertForSequenceClassification, BertTokenizerFast
# swap this for "castorini/monobert-large-msmarco" to compare against base
repo_id = "SriRamanaAtmic/AtmicReRanker"
cfg = BertConfig.from_pretrained(repo_id, num_labels=2)
model = BertForSequenceClassification.from_pretrained(repo_id, config=cfg)
tok = BertTokenizerFast.from_pretrained(repo_id) # NOT BertTokenizer — this repo has no vocab.txt,
# only tokenizer.json; BertTokenizerFast handles both
enc = tok([[query, passage]], padding=True, truncation="only_second", max_length=512, return_tensors="pt")
logits = model(**enc).logits
relevance_score = logits[:, 1] - logits[:, 0] # higher = more relevant
# or: relevance_prob = logits.softmax(-1)[:, 1]
truncation="only_second" (not truncation=True) is deliberate: it guarantees only the passage
is ever truncated, never the query, and avoids a tokenizer warning that "longest_first" (what
truncation=True maps to) raises for sequence pairs.
Training
| base model | castorini/monobert-large-msmarco |
| loss | LambdaRank-weighted pairwise: RankNet logistic loss on the relevance-score margin (logits[:,1]-logits[:,0]), scaled per pair by |ΔNDCG| from that pair's rank-swap impact |
| frozen layers | embeddings + bottom 20 of 24 BERT-large encoder layers (top 4 + pooler + classifier trainable) |
| learning rate | 2e-5 |
| batch size / grad accum | 32 / 1 (effective batch 32) |
| max sequence length | 512 |
| epochs tried | 2 |
| checkpoint selection | smallest |train_loss − val_loss| gap, checked every 50 steps (not just per-epoch) — best found at step 40, well inside epoch 1, before the gap widened further into training |
The train/val-loss-gap selection criterion (rather than best-epoch-only or a pure accuracy proxy) was chosen specifically to catch overfitting early: an earlier training run on this same data that selected checkpoints by validation pairwise-accuracy alone kept improving on that narrow metric through 4 full epochs, but regressed below the base model on the real closed-pool benchmark below — it had overfit to the training distribution's negative style in a way the accuracy proxy didn't detect. Selecting by loss gap and checking sub-epoch caught that failure mode and avoided it.
Benchmark
Closed-pool ranking on the combined layer1+layer2 validation set (1,280 queries, 1,072-passage
corpus), using SriRamanaAtmic/AtmicEmbeddingv3 as the stage-1 retriever (top-20 candidates per
query reranked; candidates outside top-20 count as a miss for every metric — stage1_ceil below is
the ceiling this caps both rerankers at).
| Metric | stage1 dense alone | base MonoBERT | AtmicReRanker (this model) |
|---|---|---|---|
| recall@1 | 0.3406 | 0.4766 | 0.4961 |
| recall@5 | 0.5508 | 0.6336 | 0.6461 |
| mrr@10 | 0.4324 | 0.5450 | 0.5604 |
| ndcg@10 | 0.4794 | 0.5787 | 0.5902 |
stage1_ceil (top-20 recall, caps both rerankers): 0.7156
This fine-tune improves on base MonoBERT across every metric on this domain's validation set.