44b-reranker-gte-modernbert

🏆 Correct document first 93% of the time, in the top 3 a remarkable 98.2%

What that means, all measured on the same test set:

• 93% vs 68% over OpenAI's general embeddings on top-1 accuracy (+25 points); 95.6% vs 74.5% on MRR.
• 93% vs 86% over BM25 keyword search; 93% vs 73% over hybrid retrieval.
• Right document in the top 3: 98.2% — up from 92.6% for the general-purpose base.
• 57% fewer ranking errors than the base it is fine-tuned from (16.7% → 7.1%). Mean rank 1.148 of 9.

On a held-out test of 3,157 queries over 1,534 scientific papers (9-way reranking with hard negatives), this 149M-parameter model picks the correct document first 93% of the time — and lands it in the top 3 a remarkable 98.2% of the time — outscoring OpenAI's general embeddings, keyword search, and hybrid retrieval, at 41 ms/query on hardware you can self-host.

Scope: reranking numbers — selecting the right document from a candidate set — not open-corpus search across millions. A companion 22.7M model (44b-reranker-minilm-l6) trades ~6 points of accuracy for ~6× the speed.

44b-reranker-gte-modernbert

A cross-encoder reranker fine-tuned on peer review → paper pairs mined from the 44B corpus.

The supervision is unusual and it is the point: the query is a peer reviewer's own summary of a paper, written by a domain expert who had never heard of this model, and the positive is the paper that review is attached to. The relevance label is not a heuristic, not a click, and not LLM-generated.

Fine-tuning on it cut the base model's error rate by 57% on a held-out, paper-disjoint evaluation.

Results

The benchmark — query and candidate IDs, and every arm's scored output — is public: NYSgpt/44b-eval.

Evaluated on 3,157 held-out queries over 1,534 papers, none of which appear anywhere in training — not as positives and not as negatives. Each query is scored against 9 candidates: 1 positive and 8 mined hard negatives. Random chance is P@1 = 0.1111.

Error rate 16.66% → 7.10%, a 57.4% relative reduction. At n = 3,157 the gap is far beyond noise (z ≈ 11.9 on an unpaired standard error, which is conservative — these are paired measurements on identical queries).

For reference, the first-stage retrieval arms scored on the same 3,157 queries:

A sibling model, NYSgpt/44b-reranker-mini, was trained on the identical data and reaches P@1 0.8654 at 7.4 ms/query — 1/7th the parameters and ~6× the speed. If latency matters more than the last three points of accuracy, use that one.

Read the eval shape before you read the numbers

This is a 9-way ranking task, not open-corpus retrieval. A 0.9290 here means "picks the right paper out of nine confusable ones," not "finds the right paper in three million." The negatives are hard by construction — nearest neighbours in embedding space plus lexical near-matches — but the pool is small and the task is closed.

What it was trained on

Splits are disjoint by paper, not by pair. Papers carry ~4.5 reviews each, so splitting by pair would place the same paper in train and test and inflate every number downstream. Proven at build time rather than asserted: |train ∩ val| = 0, |train ∩ test| = 0, |val ∩ test| = 0.

Held-out papers are excluded from the negative pool as well as the positive one. Label leakage was measured after the fact, not assumed: 0 of 1,534 test positives appear as training positives; 23 (1.5%) appear as a negative for some other query, which biases mildly against this model rather than for it.

Training

Limitations

Stated plainly, because most of these are the kind of thing a reader would otherwise have to discover.

• Query register is long-form prose. Training and evaluation queries average ~700 characters (~174 tokens) of fluent reviewer writing. This model has never seen a short keyword query. Do not assume the gains transfer to a two-word search box — that is untested.
• One discipline, one document type. Every review comes from top-tier machine learning conferences (ICLR, NeurIPS; 2024–2025). No other field, venue tier, or document class is represented.
• Context is 512, not 8192. The base supports long context; this fine-tune only ever saw 512 tokens, so the extra capacity is untrained.
• Negatives skew semantic. 92% of mined negatives came from the embedding arm and 8% from the lexical arm — the lexical miner conjoined a review's rarest terms and returned nothing on most queries, so the semantic arm backfilled. A balanced mine is the obvious next version.
• Trained on 7 negatives, evaluated against 8. Evaluation is marginally harder than training was.
• The evaluation shares a pipeline with the training data. Same source, same construction, disjoint only by paper. It measures the reranker's job honestly; it is not an independent measure of end-to-end answer quality in a live system.
• CPU latency is unmeasured. Every timing here is A10G at batch 128. No CPU serving number exists yet.
• False-negative screening may under-catch. Candidate negatives were rejected for sharing the positive's Semantic Scholar id or a near-identical title; the rate was 0.118%, and the S2-duplicate check fired zero times across 28,798 candidates. That is either a clean corpus or a check that is not working, and we have not yet established which.
• The supervision is not exhausted. This model trained on 30,509 of 77,147 available pairs. It is a first cut, not a ceiling.

Intended use

Second-stage reranking of scientific-document retrieval — reordering a first-stage candidate list for queries written in the register of a paragraph of technical prose.

Out of scope

Short keyword queries · non-ML disciplines · any setting requiring a measured CPU latency guarantee · any claim of quality on open-corpus retrieval, which this evaluation does not test.

Provenance

Reviews from OpenReview; papers from arXiv, OpenAlex and Semantic Scholar, unified in the 44B corpus. Trained 2026-07-31.

The NYSgpt model family