Cartographer : A Lightweight E-Commerce Reranker
Cartographer is a small, fast cross-encoder that scores how well a shopping query
matches a product. It takes a (query, product) pair and returns a single relevance
score for reranking a candidate list. It is a drop-in upgrade for the general-purpose
reranker it is built on: same size, same speed, measurably better on product search.
It was fine-tuned from cross-encoder/ms-marco-MiniLM-L-6-v2
on Amazon's public Shopping Queries Dataset (ESCI). The base model was trained on
web-search passages; product listings are not web text, so a domain-tuned model of the
same 22.7M-parameter footprint does meaningfully better.
Author / Co-author:
Author : 2013khansohail (Sohail Khan) Co-Author : najibkhanlive (Najibur Rahman Khan)
Results
Evaluated on the held-out ESCI test split (US / English): 8,956 queries, 181,701
(query, product) pairs. Both models scored at max_length=256, identical settings :
the comparison is apples-to-apples. The headline metric is nDCG (graded relevance), which
is what the E/S/C/I labels are for.
| Metric | Baseline (ms-marco-MiniLM-L6) | Cartographer | Δ abs | Δ rel |
|---|---|---|---|---|
| nDCG@5 | 0.6748 | 0.7272 | +0.0524 | +7.76% |
| nDCG@10 | 0.7193 | 0.7642 | +0.0449 | +6.24% |
| nDCG@20 | 0.8064 | 0.8354 | +0.0290 | +3.60% |
The gain is largest at the top of the ranking (nDCG@5), which is where it matters most for product search : users see the first few results. There were no regressions: every reported cutoff improved.
Why it works: it learned the relevance gradation
ESCI labels relevance on four graded levels : Exact, Substitute, Complement, Irrelevant : and the model was trained to regress onto graded relevance, not a binary relevant/not flag. The effect is visible in how cleanly the model separates the four classes. Mean model score per true label on the test set:
| True label | Baseline mean (± std) | Cartographer mean (± std) |
|---|---|---|
| E (Exact) | 0.024 (± 6.14) | 0.283 (± 1.27) |
| S (Substitute) | −2.228 (± 5.81) | −0.648 (± 1.02) |
| C (Complement) | −0.787 (± 6.09) | −0.781 (± 1.04) |
| I (Irrelevant) | −5.558 (± 5.93) | −1.140 (± 1.11) |
The baseline's scores are noise-dominated : its per-class standard deviation (~6) is larger than the gaps between class means, so it barely separates the labels relative to its own spread. Cartographer collapses that noise to std ≈ 1 while producing a clean, monotonic E > S > C > I ordering. That tightening is the model learning to rank product relevance.
Intended use
A second-stage reranker: retrieve a candidate set of products with a fast first-stage system (BM25 or a bi-encoder), then rerank the top-k with Cartographer for sharper ordering. It is designed to be cheap to serve at query time, which is exactly why lightweight cross-encoders get deployed in production.
Usage
from sentence_transformers import CrossEncoder
model = CrossEncoder("2013khansohail/cartographer-ecommerce-reranker-MiniLM-L6-v2", max_length=256)
query = "wireless noise cancelling headphones"
products = [
"Sony WH-1000XM5 Wireless Noise Cancelling Headphones",
"Wired Earbuds with Microphone, 3.5mm",
"Headphone Stand Aluminum Desk Holder",
"Bluetooth Speaker Waterproof Portable",
]
scores = model.predict([(query, p) for p in products])
ranked = sorted(zip(products, scores), key=lambda x: x[1], reverse=True)
for product, score in ranked:
print(f"{score:.3f} {product}")
Training
- Base model:
cross-encoder/ms-marco-MiniLM-L-6-v2(22.7M params). Fine-tuning from a reranker checkpoint rather than a raw backbone converges faster and higher on a fixed dataset. - Data: ESCI Shopping Queries, US/English,
small_versionsplit. Product text is the product title (title-only recipe for v1). - Objective: pointwise regression (MSE with a sigmoid activation) onto graded relevance gains mapped as E=1.0, S=0.1, C=0.01, I=0.0 : Amazon's ranking-task convention. The gains and the model's sigmoid-bounded output share the same [0, 1] scale.
- Recipe: 2 epochs with early stopping on validation nDCG@10; best checkpoint loaded at the end. fp16, effective batch size 64 (16 × 4 gradient accumulation), lr 2e-5, on a single consumer GPU (NVIDIA RTX 3050, 4GB).
On training length (an honest finding): one epoch reaches ~99% of the final quality (val nDCG@10 ≈ 0.759). A second epoch with early stopping adds ~0.4 nDCG@10, the gain transfers cleanly to the sealed test set (not val overfitting), and it specifically resolves the Substitute-vs-Complement ordering that one epoch leaves tangled. So 2 epochs is the shipped recipe : but one epoch is a defensible, cheaper alternative.
Limitations
- Amazon distribution. ESCI is Amazon data, so the model inherits Amazon's product distribution and category mix. It may need further tuning to transfer to other catalogs.
- Substitute vs. Complement is the soft boundary. The model orders S above C directionally, but S and C remain the closest pair and their score distributions overlap (std ≈ 1, means ~0.13 apart). This is expected: the gain mapping places both near zero, so MSE has little incentive to push them apart. If distinguishing substitutes from complements is critical for your use case, a different gain mapping or a ranking loss is the place to look.
- Lightweight ceiling. This is the best model in its weight class, not the best model available. Some hard queries will need a heavier reranker. The pitch is "drop-in upgrade at the same cost," not "state of the art."
- English-only, title-only product text in this version.
Citation
Built on the Amazon Shopping Queries Dataset (ESCI):
Reddy et al., Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search (2022). https://github.com/amazon-science/esci-data
Base model: cross-encoder/ms-marco-MiniLM-L-6-v2.
Training code, data prep, and full evaluation (reproducible end to end): github.com/sohail000/cartographer-reranker