CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased on the msmarco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text
• Training Dataset:
  • msmarco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("kwondw/reranker-MiniLM-L12-H384-uncased-msmarco-margin-mse")
# Get scores for pairs of inputs
pairs = [
    ['cost for laminate flooring per square foot', 'Average Price Range. Cheap laminate flooring starts at less than $1 per square foot and can cost as much as $5 per square foot. The average laminate flooring costs from $2.40-$4 per square foot.The price is determined, as it is with most products, by the quality of the material.A major part of that quality is the thickness of the wear layer.: There will be some variation in prices regionally and the final amount will depend on the size and complexity of the project, that said the average cost to install laminate flooring is $1.40 to $2.20 per square foot.'],
    ['what county is south daytona, fl in?', 'South Daytona, Florida. South Daytona is a city in Volusia County, Florida, United States. The population was 12,252 at the 2010 census.'],
    ['when was donkey kong released', 'Donkey Kong (ã\x83\x89ã\x83³ã\x82\xadã\x83¼ã\x82³ã\x83³ã\x82°, DonkÄ« Kongu, [dõÅ\x8b.kiË\x90 kõÅ\x8b.É¡uÍ\x8d]) is a series of video games featuring the adventures of a gorilla character called Donkey Kong, conceived by Shigeru Miyamoto in 1981. The franchise mainly comprises two different game genres, plus spin-off titles of various genres.'],
    ['causes of compartment syndrome of tibia', 'Both acute exertional compartment syndrome and anterior tibial stress fractures are rare causes of leg pain in young athletes. As demonstrated by this case, their combined presentation can pose a diagnostic dilemma.'],
    ['how many years of college to be a dental hygienist', "Education and Qualifications. To become a dental hygienist, you'll need either a 2- or 4-year degree in dental hygiene. Most programs are two years long and lead to an associate's degree. They are offered through community colleges and technical colleges."],
]
scores = model.predict(pairs)
print(scores)
# [1.     1.     0.9998 0.991  1.    ]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'cost for laminate flooring per square foot',
    [
        'Average Price Range. Cheap laminate flooring starts at less than $1 per square foot and can cost as much as $5 per square foot. The average laminate flooring costs from $2.40-$4 per square foot.The price is determined, as it is with most products, by the quality of the material.A major part of that quality is the thickness of the wear layer.: There will be some variation in prices regionally and the final amount will depend on the size and complexity of the project, that said the average cost to install laminate flooring is $1.40 to $2.20 per square foot.',
        'South Daytona, Florida. South Daytona is a city in Volusia County, Florida, United States. The population was 12,252 at the 2010 census.',
        'Donkey Kong (ã\x83\x89ã\x83³ã\x82\xadã\x83¼ã\x82³ã\x83³ã\x82°, DonkÄ« Kongu, [dõÅ\x8b.kiË\x90 kõÅ\x8b.É¡uÍ\x8d]) is a series of video games featuring the adventures of a gorilla character called Donkey Kong, conceived by Shigeru Miyamoto in 1981. The franchise mainly comprises two different game genres, plus spin-off titles of various genres.',
        'Both acute exertional compartment syndrome and anterior tibial stress fractures are rare causes of leg pain in young athletes. As demonstrated by this case, their combined presentation can pose a diagnostic dilemma.',
        "Education and Qualifications. To become a dental hygienist, you'll need either a 2- or 4-year degree in dental hygiene. Most programs are two years long and lead to an associate's degree. They are offered through community colleges and technical colleges.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Training Details

Training Dataset

msmarco

• Dataset: msmarco at 2a16a97
• Size: 500,000 training samples
• Columns: score, query, positive, and negative
• Approximate statistics based on the first 100 samples:

  	score	query	positive	negative
  type	float	string	string	string
  modality		text	text	text
  details	min: 1.61mean: 13.55max: 21.63	min: 4 tokensmean: 8.98 tokensmax: 13 tokens	min: 24 tokensmean: 81.15 tokensmax: 237 tokens	min: 23 tokensmean: 78.24 tokensmax: 171 tokens

• Samples:

  score	query	positive	negative
  6.981858521699905	symptoms of shingles on back	The earliest symptoms of shingles, which include headache, fever, and malaise, are nonspecific, and may result in an incorrect diagnosis. These symptoms are commonly followed by sensations of burning pain, itching, hyperesthesia (oversensitivity), or paresthesia (pins and needles: tingling, pricking, or numbness).	Shingles is caused by the varicella-zoster virus, the virus that also causes chickenpox. In a person who has been exposed to chickenpox -- or its vaccine -- the virus never really goes away.It can lie dormant in the body's nerves.f you have shingles symptoms, get treatment now and you may avoid permanent nerve pain. Shingles, a viral infection of the nerve roots, affects 1 million people in the U.S each year. Most people recover from their bout, but for as many as 50% of those over age 60 who have not been treated, the pain doesn't go away.
  18.709746996561687	uuww airport	Vnukovo International Airport. Vnukovo International Airport (Russian: международный аэропорт внуково; IPA: [ˈvnukəvə]) (IATA: VKO, ICAO: UUWW), is a dual-runway international airport located 28 kilometres (17 mi) southwest of the centre of Moscow, Russia.	Javascript is currently disabled in your browser. Arrivals& Departures Timetable Budgetflightsout ofBrussels Airport To & fromBrussels Airport Parking at the airport Worldwidebudget flights &city trips Informationon air trafficto & fromBrussels Airport Shops,bars &restaurants Baggage&TravelInfo.avascript is currently disabled in your browser. Arrivals& Departures Timetable Budgetflightsout ofBrussels Airport To & fromBrussels Airport Parking at the airport Worldwidebudget flights &city trips Informationon air trafficto & fromBrussels Airport Shops,bars &restaurants Baggage&TravelInfo.
  9.210955460866293	what is a hepatitis b surface antibody test	This test looks for hepatitis B surface antigens in your blood. The test is used to find out whether you have a recent or long-standing infection from the hepatitis B virus (HBV). HBV has proteins called antigens on its surface that cause your immune system to make antibodies.	Blood tests for hepatitis C include the following: 1 Screening test for antibodies to the hepatitis C virus. A screening blood test will show whether you have developed antibodies to the hepatitis C virus. A positive antibody test means you were exposed to the hepatitis C virus at some point.

• Loss: MarginMSELoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity"
  }
  

Evaluation Dataset

msmarco

• Dataset: msmarco at 2a16a97
• Size: 10,000 evaluation samples
• Columns: score, query, positive, and negative
• Approximate statistics based on the first 100 samples:

  	score	query	positive	negative
  type	float	string	string	string
  modality		text	text	text
  details	min: -1.07mean: 13.75max: 21.54	min: 5 tokensmean: 9.12 tokensmax: 18 tokens	min: 26 tokensmean: 78.65 tokensmax: 166 tokens	min: 25 tokensmean: 73.67 tokensmax: 176 tokens

• Samples:

  score	query	positive	negative
  17.243099629878998	cost for laminate flooring per square foot	Average Price Range. Cheap laminate flooring starts at less than $1 per square foot and can cost as much as $5 per square foot. The average laminate flooring costs from $2.40-$4 per square foot.The price is determined, as it is with most products, by the quality of the material.A major part of that quality is the thickness of the wear layer.: There will be some variation in prices regionally and the final amount will depend on the size and complexity of the project, that said the average cost to install laminate flooring is $1.40 to $2.20 per square foot.	We created the most comprehensive flooring installation calculator on the internet. We’ve broken it down to the following: Step 1 – Laminate Type. Step 2 – Location, Square Footage, Materials, and Labor. Step 3 – Old Flooring and New Flooring. Step 4 – Get your calculation. Put it all together and we give you the most accurate flooring calculation available.
  15.025473694006603	what county is south daytona, fl in?	South Daytona, Florida. South Daytona is a city in Volusia County, Florida, United States. The population was 12,252 at the 2010 census.	Driving distance from Ocala, FL to Daytona Beach, FL. The total driving distance from Ocala, FL to Daytona Beach, FL is 78 miles or 126 kilometers. Your trip begins in Ocala, Florida. It ends in Daytona Beach, Florida.
  19.3864418665568	when was donkey kong released	Donkey Kong (ドンキーコング, Donkī Kongu, [dõŋ.kiː kõŋ.ɡu͍]) is a series of video games featuring the adventures of a gorilla character called Donkey Kong, conceived by Shigeru Miyamoto in 1981. The franchise mainly comprises two different game genres, plus spin-off titles of various genres.	• Overfeeding a donkey or feeding it the wrong diet can be fatal. • Overfeeding can result a crippling disease called laminitis of the hooves. • Obesity is also a consequence of overfeeding. • Once obese it is difficult to get a donkey back to normal weight because they require so few calories to sustain themselves.

• Loss: MarginMSELoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• learning_rate: 8e-06
• warmup_steps: 0.1
• fp16: True
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• seed: 12
• dataloader_num_workers: 2
• dataloader_persistent_workers: True

All Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 8e-06
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: True
• dataloader_persistent_workers: True
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.4 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.0
• Transformers: 5.13.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}