exoldoff/bge-reranker-v2-m3-cross-encoder-marketplaces-rus

Fine-tuned cross-encoder reranker for Russian marketplace SKU deduplication.

The model is based on BAAI/bge-reranker-v2-m3 and was fine-tuned to score whether two marketplace product records describe the same base SKU.

It is intended for pairwise product matching / entity resolution pipelines where candidate pairs are generated first, then this model reranks or classifies those pairs.

Critical Scope

This model is intentionally narrow. It was built for Russian marketplace catalog data, not for general semantic ranking or universal product matching.

• Marketplaces: Ozon, Yandex Market / Яндекс.Маркет, Wildberries / WB / ВБ.
• Supported categories: Sauces / Соусы / Соус, Coconut oil / Кокосовое масло, Soap / Мыло.
• Language/domain: Russian and mixed Russian marketplace product titles with structured SKU fields such as brand, category, marketplace, unit weight, total weight, and pack count.

Use outside these marketplaces or categories should be treated as experimental and must be recalibrated on target data before production use.

Intended Use

Given two structured product texts from the supported Russian marketplace categories, the model returns a score where a higher value means the pair is more likely to be the same base product.

The training target treats pack-count variants as positive when the underlying product is the same. For example, 1 x 500 ml and 2 x 500 ml may still be considered the same base SKU. Final pack-level grouping should be handled downstream with deterministic pack rules.

Typical pipeline:

1. Generate candidate product pairs within one supported category using lexical or embedding retrieval.
2. Score candidate pairs with this cross-encoder.
3. Select a threshold on a dev split.
4. Build product-family edges from pairs above the selected threshold.
5. Apply deterministic pack/grouping logic after pairwise matching.

Model Details

• Model type: Sentence Transformers CrossEncoder
• Base model: BAAI/bge-reranker-v2-m3
• Fine-tuned model id: exoldoff/bge-reranker-v2-m3-cross-encoder-marketplaces-rus
• Task: binary same-base-SKU scoring for marketplace product pairs
• Output: sigmoid probability-like score
• Max sequence length during training: 192 tokens
• Loss: BinaryCrossEntropyLoss
• Training precision: fp16

Training Data

The model was trained on a frozen multi-category Russian marketplace SKU deduplication dataset.

Training scope:

• Marketplaces: Ozon, Yandex Market / Яндекс.Маркет, Wildberries / WB / ВБ.
• Categories: Sauces / Соусы / Соус, Coconut oil / Кокосовое масло, Soap / Мыло.

Rows used by the training run:

Input columns were converted into structured pair texts such as:

brand: ... | title: ... | subcategory: ... | marketplace: ... | unit_weight_kg: ... | total_weight_kg: ... | pack_count: ...

Labels:

• 1: same base product / exact duplicate for the ML layer
• 0: different product

Evaluation

Metrics below are from the training manifest at the default 0.5 prediction threshold. In production or research comparison, select the operating threshold on a dev split according to the business cost of errors.

For SKU deduplication, false merges are usually more expensive than missed duplicates. Prefer cost-sensitive threshold calibration over blindly using 0.5.

Benchmark Comparison

The table below compares the selected BGE fine-tuned reranker with the main fine-tuned and zero-shot alternatives on the frozen SKU deduplication test split. The report uses thresholds selected on the dev split with the threshold_weighted_cost strategy.

Higher precision, recall, and F1 are better. Lower false merges and weighted cost are better. Weighted cost weights pair errors by available sales volume.

Compared with the zero-shot BGE reranker baseline on the main test split:

• Weighted cost improved from 477.8 to 131.3.
• Recall improved from 0.436 to 0.795.
• False merges decreased from 4 to 0.

Freeze: 2465 binary pairs from the multi-category labeling set; training-ready split 1404 / 167 / 165.

Usage

Install:

pip install -U sentence-transformers

Run pair scoring:

from sentence_transformers import CrossEncoder

model = CrossEncoder("exoldoff/bge-reranker-v2-m3-cross-encoder-marketplaces-rus")

pairs = [
    (
        "brand: heinz | title: соус томатный heinz 500 г | category: Соус | marketplace: Ozon | unit_weight_kg: 0.5 | total_weight_kg: 0.5 | pack_count: 1",
        "brand: heinz | title: томатный соус heinz 500г | category: Соусы | marketplace: Яндекс.Маркет | unit_weight_kg: 0.5 | total_weight_kg: 0.5 | pack_count: 1",
    ),
    (
        "brand: example | title: мыло жидкое алоэ 500 мл | category: Мыло | marketplace: WB | unit_weight_kg: 0.5 | total_weight_kg: 0.5 | pack_count: 1",
        "brand: example | title: мыло туалетное детское 90 г | category: Мыло | marketplace: Ozon | unit_weight_kg: 0.09 | total_weight_kg: 0.09 | pack_count: 1",
    ),
]

scores = model.predict(pairs)
print(scores)

For explicit sigmoid output in newer Sentence Transformers versions:

import torch
from sentence_transformers import CrossEncoder

model = CrossEncoder("exoldoff/bge-reranker-v2-m3-cross-encoder-marketplaces-rus")
scores = model.predict(pairs, activation_fn=torch.nn.Sigmoid())

Limitations

• This model is only trained and documented for Russian marketplace SKU deduplication on Ozon, Yandex Market / Яндекс.Маркет, and Wildberries / WB / ВБ.
• Supported categories are limited to Sauces / Соусы / Соус, Coconut oil / Кокосовое масло, and Soap / Мыло.
• This model is a reranker/classifier for candidate pairs, not a candidate generator.
• It is not a universal product matcher and is not validated for other marketplace domains, countries, languages, or categories.
• Scores should be calibrated on the target catalog and business objective.
• The model predicts same base product, not final pack identity. Pack-size resolution should happen downstream.
• The dataset is small enough that category-specific blind spots are possible.

Training Configuration

Selected hyperparameters:

Runtime package versions from the manifest:

Provenance

• Base model: BAAI/bge-reranker-v2-m3
• Fine-tuned model id: exoldoff/bge-reranker-v2-m3-cross-encoder-marketplaces-rus