CrossEncoder based on answerdotai/ModernBERT-base

This is a Cross Encoder model finetuned from answerdotai/ModernBERT-base on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text
• Training Dataset:
  • ms_marco

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'ModernBertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("kwondw/reranker-msmarco-v1.1-ModernBERT-base-bce")
# Get scores for pairs of inputs
pairs = [
    ['define ransom', 'Full Definition of RANSOM. transitive verb. 1. : to deliver especially from sin or its penalty. 2. : to free from captivity or punishment by paying a price. — ran·som·er noun. See ransom defined for English-language learners. rescue, deliver, redeem, ransom, reclaim, save mean to set free from confinement or danger. rescue implies freeing from imminent danger by prompt or vigorous action.'],
    ['what does atlas copco do', 'Atlas Copco is a Swedish industrial company that was founded in 1873. It manufactures industrial tools and equipment. The Atlas Copco Group is a global industrial group of companies headquartered in Nacka, Sweden. In 2014, global revenues totalled SEK 94 billion, and the company employed more than 44,000 people. 1 2009 – Atlas Copco is the first company to get a compressor certified for 100 percent energy recovery by the TUEV tüv (Institute German Technical Monitoring).  association'],
    ['two situations in which dna fingerprinting is useful', 'Paternity and Maternity This is also a well known use for DNA fingerprinting. This is the test used to find out who is the father of a baby or children. Every individual has a VNTR pattern which is inherited from their parents.'],
    ['sustainable business strategy definition', 'Jupiterimages/Brand X Pictures/Getty Images. A business strategy defines how a company intends to operate in its markets. As environmental awareness has grown over the recent decades, businesses of all forms are being forced to incorporate more sustainable approaches to carrying out their activities.'],
    ['how do you determine salinity', 'RSS Feeds. When we measure the salinity of water, we look at how much dissolved salt is in the water, or the concentration of salt in the water. Concentration is the amount (by weight) of salt in water and can be expressed in parts per million (ppm). Here are the classes of water:'],
]
scores = model.predict(pairs)
print(scores)
# [0.2587 0.4132 0.1375 0.1348 0.0626]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'define ransom',
    [
        'Full Definition of RANSOM. transitive verb. 1. : to deliver especially from sin or its penalty. 2. : to free from captivity or punishment by paying a price. — ran·som·er noun. See ransom defined for English-language learners. rescue, deliver, redeem, ransom, reclaim, save mean to set free from confinement or danger. rescue implies freeing from imminent danger by prompt or vigorous action.',
        'Atlas Copco is a Swedish industrial company that was founded in 1873. It manufactures industrial tools and equipment. The Atlas Copco Group is a global industrial group of companies headquartered in Nacka, Sweden. In 2014, global revenues totalled SEK 94 billion, and the company employed more than 44,000 people. 1 2009 – Atlas Copco is the first company to get a compressor certified for 100 percent energy recovery by the TUEV tüv (Institute German Technical Monitoring).  association',
        'Paternity and Maternity This is also a well known use for DNA fingerprinting. This is the test used to find out who is the father of a baby or children. Every individual has a VNTR pattern which is inherited from their parents.',
        'Jupiterimages/Brand X Pictures/Getty Images. A business strategy defines how a company intends to operate in its markets. As environmental awareness has grown over the recent decades, businesses of all forms are being forced to incorporate more sustainable approaches to carrying out their activities.',
        'RSS Feeds. When we measure the salinity of water, we look at how much dissolved salt is in the water, or the concentration of salt in the water. Concentration is the amount (by weight) of salt in water and can be expressed in parts per million (ppm). Here are the classes of water:',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco
• Size: 666,193 training samples
• Columns: query, passage, and label
• Approximate statistics based on the first 100 samples:

  	query	passage	label
  type	string	string	int
  modality	text	text
  details	min: 5 tokensmean: 9.21 tokensmax: 19 tokens	min: 39 tokensmean: 94.57 tokensmax: 163 tokens	0: ~85.58%1: ~14.42%

• Samples:

  query	passage	label
  how did the vietnam war contributed to the war powers resolution	The War Powers Act is a reaction to the Vietnam War. Congress passed it in 1973 when the United States withdrew from combat operations in Vietnam after more than a decade. The War Powers Act attempted to correct what Congress and the American public saw as excessive war-making powers in the hands of the president. Congress was also attempting to correct a mistake of its own.	0
  what is Fra Angelico contribution renaissance art	Fra Angelico Facts. Fra Angelico was an early Italian Renaissance artist and a monk, considered to be one of the most talented fresco painters of his time. He was born Guido di Pietro in approximately 1395 at Rupecanina in Mugello, but nothing is known of his parents.	1
  what is mederma made of	Mederma is made from a patented formula which combines unique ingredients such as Cepalin, centella asiatica leaf extract and hyaluronic acid. Cepalin is the main active ingredient of the product and is an onion extract that has been known to soothe the skin from inflammation and irritation. 1 While this will not do much in terms of the reduction of stretch marks, softer skin will be less likely to crack or show the wear of agedness. 2 Unfortunately, such a positive trait is not what most consumers would be primarily looking for when purchasing Mederma.	1

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco
• Size: 10,000 evaluation samples
• Columns: query, passage, and label
• Approximate statistics based on the first 100 samples:

  	query	passage	label
  type	string	string	int
  modality	text	text
  details	min: 4 tokensmean: 8.66 tokensmax: 16 tokens	min: 37 tokensmean: 93.02 tokensmax: 170 tokens	0: ~87.50%1: ~12.50%

• Samples:

  query	passage	label
  define ransom	Full Definition of RANSOM. transitive verb. 1. : to deliver especially from sin or its penalty. 2. : to free from captivity or punishment by paying a price. — ran·som·er noun. See ransom defined for English-language learners. rescue, deliver, redeem, ransom, reclaim, save mean to set free from confinement or danger. rescue implies freeing from imminent danger by prompt or vigorous action.	0
  what does atlas copco do	Atlas Copco is a Swedish industrial company that was founded in 1873. It manufactures industrial tools and equipment. The Atlas Copco Group is a global industrial group of companies headquartered in Nacka, Sweden. In 2014, global revenues totalled SEK 94 billion, and the company employed more than 44,000 people. 1 2009 – Atlas Copco is the first company to get a compressor certified for 100 percent energy recovery by the TUEV tüv (Institute German Technical Monitoring). association	1
  two situations in which dna fingerprinting is useful	Paternity and Maternity This is also a well known use for DNA fingerprinting. This is the test used to find out who is the father of a baby or children. Every individual has a VNTR pattern which is inherited from their parents.	0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_steps: 0.1
• seed: 12
• fp16: True
• load_best_model_at_end: True
• ddp_find_unused_parameters: False

All Hyperparameters

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 12
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.6 hours
• Evaluation: 20.4 minutes
• Total: 1.9 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 6.0.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}