ms-marco-MiniLM-L6-v2 fine-tuned as an Indian tax-law reranker

A cross-encoder reranker, fully fine-tuned from cross-encoder/ms-marco-MiniLM-L6-v2 for second-stage reranking over Indian tax-law judgments. It reads a (query, passage) pair together and scores how well the passage answers the question.

Used to rerank the top 50 results from bge-small-en-v1.5-indian-tax-law-lora, it improves nDCG@10 by +12.4% and Recall@10 by +11.4% over that retriever alone.

The finding you should know before using this

A zero-shot MS-MARCO reranker makes this task worse, not better. On the same shortlists, the un-tuned base model scores nDCG@10 0.4837 — below the 0.5015 you get by not reranking at all. The gain here comes from domain fine-tuning, not from adding a reranker.

A paired bootstrap over the same 256 queries and identical shortlists (10,000 resamples) puts the fine-tuned-versus-zero-shot gap at +0.0798 nDCG@10, 95% CI [+0.0516, +0.1092], p < 0.0001, better on 83 queries and worse on 33. The interval excludes zero.

So: do not substitute the base model here and expect similar behaviour.

Usage

The reranker takes bare query text — unlike the bi-encoder, it needs no prefix.

from sentence_transformers import CrossEncoder

model = CrossEncoder("FlorianTraugott/ms-marco-MiniLM-L6-v2-indian-tax-law", max_length=512)

query = "Can a reassessment notice be issued after four years?"
shortlist = ["<passage 1>", "<passage 2>", "..."]   # top-50 from the bi-encoder

scores = model.predict([(query, p) for p in shortlist])
reranked = [p for _, p in sorted(zip(scores, shortlist), key=lambda x: -x[0])]

Scores are raw logits (the model uses an identity activation), so they are unbounded and only meaningful for ordering — not as calibrated probabilities.

Results

256 held-out queries against a 3,868-passage corpus, reranking the top N=50.

Choosing the rerank depth

A reranker permutes a shortlist; it cannot add to it. So post-rerank Recall@10 can never exceed the retriever's Recall@N, and at N=10 it is provably unchanged.

N=50 is the operating point: it maximises every metric measured, and N=100 is both slower and worse. At N=10, Recall is frozen by construction while MRR@10 still rises +10.0% and nDCG@10 +7.0% on the identical ten documents — the reranker doing purely its reordering job.

Training

LoRA was tried and rejected: under task_type=SEQ_CLS, PEFT attaches without error but leaves modules_to_save=None, so the classification head stays frozen while the body adapts underneath it. Full fine-tuning was verified live instead.

Training data

Same corpus and split as the companion retriever: 3,868 passages from 400 Indian tax judgments filtered from opennyaiorg/InJudgements_dataset (Apache-2.0), with synthetic queries generated by OpenAI gpt-4.1-mini under anti-echo constraints, and 5 hard negatives per query mined with the un-tuned retriever.

The reranker trains on the same hard negatives the retriever saw, so its difficulty is calibrated to the shortlists it actually sits behind. Eval positives are excluded from training in both label slots; all leak checks measure 0.

Full pipeline and provenance: https://github.com/FlorianTraugott/legal-retrieval-finetune

Intended use and limitations

Intended for reranking a retrieval shortlist over Indian tax-law judgments, at N≈50, behind the companion retriever bge-small-en-v1.5-indian-tax-law-lora. Neither model is the full story alone: the retriever sets the recall ceiling this model reorders within.

Not intended for other legal domains or jurisdictions, general-purpose reranking, or scoring passages outside a retrieved shortlist. It is a ranking component, not a legal-advice system.

1. Reranking is net-positive, not uniformly safe. Of 221 queries with a positive in the top 50: 91 improved, 69 unchanged, 61 got worse. One positive fell from rank 1 to 22. Net top-10 swing was +20.
2. This checkpoint was selected on the evaluation set (by nDCG@10 at N=50), so its figures carry mild selection optimism. The effect is bounded — only three candidates were compared and the two trained ones differ by 0.0008 nDCG@10.
3. The in-training validation metric was saturated (0.8983 zero-shot on a 1-in-6 ranking proxy), which is why checkpoint selection was redone on the real task.
4. The eval queries are synthetic, each with a single labelled correct passage, so the attainable ceiling is below 1.0 and unknown.
5. Latency was never benchmarked. N=50 means 50 cross-encoder forward passes per query, serially, after retrieval. Batched throughput during evaluation was 139–179 pairs/second on an Apple Silicon laptop, which is not a served single-query path.
6. About 44% of the available headroom is converted: at N=50 the ceiling is Recall@10 0.8633 and this model reaches 0.7656.

Licence

Apache-2.0, inherited from cross-encoder/ms-marco-MiniLM-L6-v2 (Apache-2.0); this is a full fine-tune of those weights, so the derivative carries the same terms and its notice requirements. The training corpus derives from opennyaiorg/InJudgements_dataset (Apache-2.0). Synthetic training queries were generated using OpenAI gpt-4.1-mini; if you redistribute or build on this work, review OpenAI's terms for your own use case.

Author: Ayush Padhy (GitHub/HF: @FlorianTraugott)

Citation

@software{legal_retrieval_finetune_2026,
  title  = {Fine-tuning a two-stage retrieval stack for Indian tax law},
  year   = {2026},
  url    = {https://github.com/FlorianTraugott/legal-retrieval-finetune}
}