44b-reranker-minilm-l6

🏆 22.7M model that beats the generalists

A cross-encoder reranker fine-tuned on peer review → paper pairs mined from the 44B corpus.

The query is a peer reviewer's own summary of a paper, written by a domain expert who had never heard of this model, and the positive is the paper that review is attached to.

The relevance label is not a heuristic, not a click, and not LLM-generated. Fine-tuning on it cut the base model's error rate by 62% — and this 22M-parameter model, at 7.4 ms/query, then outscores a 150M-parameter reranker running off the shelf. On a held-out test of 3,157 queries over scientific papers (9-way reranking with hard negatives), this 22.7M-parameter model — small enough to run almost anywhere, at 7.4 ms/query — picks the right document first 87% of the time and lands it in the top 3 over 95% of the time, beating OpenAI's general embeddings and hybrid retrieval and edging keyword search.

What that means, all measured on the same test set:

• 87% vs 68% over OpenAI's general embeddings on top-1 accuracy (+19 points); 91.4% vs 74.5% on MRR.
• Right document in the top 3: 95.4%.
• Beats hybrid retrieval 87% vs 73%, and edges BM25 — at a fraction of the size.
• 62% fewer ranking errors than the base it is fine-tuned from (35.7% → 13.5%). Mean rank 1.322 of 9, 7.4 ms/query.

*Scope: reranking numbers — selecting the right document from a candidate set — not open-corpus search.

• Also see: 44b-reranker-gte-modernbert reaches 93% at 149.6M params.*

Results

The benchmark — query and candidate IDs, and every arm's scored output — is public: NYSgpt/44b-eval.

Evaluated on 3,157 held-out queries over 1,534 papers, none of which appear anywhere in training — not as positives and not as negatives. Each query is scored against 9 candidates: 1 positive and 8 mined hard negatives. Random chance is P@1 = 0.1111.

Error rate 35.73% → 13.46%, a 62.3% relative reduction.

At n = 3,157 the gap is far beyond noise (z ≈ 21.3 on an unpaired standard error, which is conservative — these are paired measurements on identical queries).

The size comparison is the interesting one. gte-reranker-modernbert-base — 150M parameters, off the shelf — scores 0.8334 at 53.1 ms/query on this same evaluation. This model beats it on accuracy at roughly 7× the speed and 1/7th the parameters. Domain supervision bought more than 7× the model did.

For reference, the first-stage retrieval arms scored on the same 3,157 queries:

NYSgpt/44b-reranker Sibling model was trained on identical data and reaches P@1 0.9290 at 41.1 ms/query.

What it was trained on

Splits are disjoint by paper, not by pair. Papers carry ~4.5 reviews each, so splitting by pair would place the same paper in train and test and inflate every number downstream. Proven at build time rather than asserted: |train ∩ val| = 0, |train ∩ test| = 0, |val ∩ test| = 0.

Held-out papers are excluded from the negative pool as well as the positive one. Label leakage was measured after the fact, not assumed: 0 of 1,534 test positives appear as training positives; 23 (1.5%) appear as a negative for some other query, which biases mildly against this model rather than for it.

Training

Limitations

• Query register is long-form prose. Training and evaluation queries average ~700 characters (~174 tokens) of fluent reviewer writing. This model has never seen a short keyword query. Do not assume the gains transfer to a two-word search box — that is untested.
• One discipline, one document type. Every review comes from top-tier machine learning conferences (ICLR, NeurIPS; 2024–2025). No other field, venue tier, or document class is represented.
• Short context is a real constraint for this task. Queries are ~180-token review summaries against long abstracts; at 256 tokens most pairs are truncated. The accuracy above is achieved despite that, which is part of why the fine-tune matters so much here.
• Negatives skew semantic. 92% of mined negatives came from the embedding arm and 8% from the lexical arm — the lexical miner conjoined a review's rarest terms and returned nothing on most queries, so the semantic arm backfilled. A balanced mine is the obvious next version.
• Trained on 7 negatives, evaluated against 8. Evaluation is marginally harder than training was.
• The evaluation shares a pipeline with the training data. Same source, same construction, disjoint only by paper. Not an independent measure of end-to-end answer quality in a live system.
• CPU latency is unmeasured. Every timing here is A10G at batch 128. No CPU serving number exists yet.
• False-negative screening may under-catch. Candidate negatives were rejected for sharing the positive's Semantic Scholar id or a near-identical title; the rate was 0.118%, and the S2-duplicate check fired zero times across 28,798 candidates.
• The supervision is not exhausted. This model trained on 30,509 of 77,147 available pairs. V2 pending.

Eval shape

This is a 9-way ranking task, not open-corpus retrieval. A 0.8654 here means "picks the right paper out of nine confusable ones," not "finds the right paper in three million." The negatives are hard by construction — nearest neighbours in embedding space plus lexical near-matches — but the pool is small and the task is closed.

Intended use

Second-stage reranking of scientific-document retrieval — reordering a first-stage candidate list for queries written in the register of a paragraph of technical prose. The speed profile suits a live request path.

Out of scope

Short keyword queries · non-ML disciplines · any setting requiring a measured CPU latency guarantee · any claim of quality on open-corpus retrieval, which this evaluation does not test.

Provenance

Reviews from OpenReview; papers from arXiv, OpenAlex and Semantic Scholar, unified in corpus. Trained 2026-07-31.

The NYSgpt model family