CrossEncoder based on jhu-clsp/ettin-encoder-68m

This is a Cross Encoder model finetuned from jhu-clsp/ettin-encoder-68m on the msmarco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: jhu-clsp/ettin-encoder-68m
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • msmarco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("tomaarsen/ms-marco-ettin-68m-reranker")
# Get scores for pairs of texts
pairs = [
    ['does a cars registered owner have to be the insurance policy holder?', 'For example when a person has a company car they may be registered as the keeper though they are not the owner. Named driver-this is a driver who has been added to a car insurance policy to be an additional one to the main driver.'],
    ['how to pronounce lucretia', "Lucretia /lu-cre-tia/ [3 sylls.] as a girls' name is pronounced loo-KREE-shah. It is of Latin origin, and the meaning of Lucretia is succeed. Also possibly of Etruscan origin and of uncertain meaning. The name of a Roman matron who committed suicide in public protest against dishonor."],
    ['average cost of gym equipment', 'When equipping a home-gym, itâ\x80\x99s critical to consider at least one piece of cardio equipment. The top four treadmill picks from Consumer Search range from $990 to $3,500, with the average costing around $2,000.Another piece of popular cardio equipment is an elliptical trainer, which averages $1,500.he top four treadmill picks from Consumer Search range from $990 to $3,500, with the average costing around $2,000. Another piece of popular cardio equipment is an elliptical trainer, which averages $1,500.'],
    ['is cerebral palsy a disease', 'It occurs in about 2.1 per 1,000 live births. Cerebral palsy has been documented throughout history with the first known descriptions occurring in the work of Hippocrates in the 5th century BCE. Extensive study of the condition began in the 19th century by William John Little, after whom it was called Little disease.'],
    ['what is a service in cherwell', 'Cherwell Service Management (CSM) is a cloud-based software product that helps an IT organization to deliver certified ITIL processes such as: Additionally, it has the ability to provide a customer-friendly IT self-service portal, powerful dashboards and reporting.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'does a cars registered owner have to be the insurance policy holder?',
    [
        'For example when a person has a company car they may be registered as the keeper though they are not the owner. Named driver-this is a driver who has been added to a car insurance policy to be an additional one to the main driver.',
        "Lucretia /lu-cre-tia/ [3 sylls.] as a girls' name is pronounced loo-KREE-shah. It is of Latin origin, and the meaning of Lucretia is succeed. Also possibly of Etruscan origin and of uncertain meaning. The name of a Roman matron who committed suicide in public protest against dishonor.",
        'When equipping a home-gym, itâ\x80\x99s critical to consider at least one piece of cardio equipment. The top four treadmill picks from Consumer Search range from $990 to $3,500, with the average costing around $2,000.Another piece of popular cardio equipment is an elliptical trainer, which averages $1,500.he top four treadmill picks from Consumer Search range from $990 to $3,500, with the average costing around $2,000. Another piece of popular cardio equipment is an elliptical trainer, which averages $1,500.',
        'It occurs in about 2.1 per 1,000 live births. Cerebral palsy has been documented throughout history with the first known descriptions occurring in the work of Hippocrates in the 5th century BCE. Extensive study of the condition began in the 19th century by William John Little, after whom it was called Little disease.',
        'Cherwell Service Management (CSM) is a cloud-based software product that helps an IT organization to deliver certified ITIL processes such as: Additionally, it has the ability to provide a customer-friendly IT self-service portal, powerful dashboards and reporting.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Training Details

Training Dataset

msmarco

• Dataset: msmarco at 9e329ed
• Size: 39,770,704 training samples
• Columns: query_id, positive_id, negative_id, and score
• Approximate statistics based on the first 1000 samples:

  	query_id	positive_id	negative_id	score
  type	string	string	string	float
  details	min: 9 charactersmean: 34.31 charactersmax: 135 characters	min: 52 charactersmean: 356.08 charactersmax: 843 characters	min: 75 charactersmean: 350.32 charactersmax: 990 characters	min: -3.52mean: 13.57max: 22.49

• Samples:

  query_id	positive_id	negative_id	score
  what is the salary for a supervisor at tyson foods?	Tyson Foods Production Supervisor Salary. Tyson Foods Production Supervisor average salary is $38,288, median salary is $36,000 with a salary range from $31,000 to $71,200. Tyson Foods Production Supervisor salaries are collected from government agencies and companies. Each salary is associated with a real job position. Tyson Foods Production Supervisor salary statistics is not exclusive and is for reference only. They are presented as is and updated regularly.	Internal Reporting. An employee should follow an employer's policy for making a report of discrimination. In general, the employee should report the discrimination to her supervisor. If the supervisor is responsible for the discrimination, the employee should report the incident to the supervisor's supervisor.	21.560779412587486
  what is a vulva	The vulva is the outer part of the female genitals. The vulva includes the opening of the vagina (sometimes called the vestibule), the labia majora (outer lips), the labia minora (inner lips), and the clitoris.Around the opening of the vagina, there are 2 sets of skin folds.he Bartholin glands are found just inside the opening of the vagina -- one on each side. These glands produce a mucus-like fluid that acts as a lubricant during sex. At the front of the vagina, the labia minora meet to form a fold or small hood of skin called the prepuce.	Vulvar cancer is a type of cancer that occurs on the outer surface area of the female genitalia. The vulva is the area of skin that surrounds the urethra and vagina, including the clitoris and labia. Vulvar cancer commonly forms as a lump or sore on the vulva that often causes itching. Though it can occur at any age, vulvar cancer is most commonly diagnosed in older women. Vulvar cancer treatment usually involves surgery to remove the cancer and a small amount of surrounding healthy tissue. Sometimes vulvar cancer surgery requires removing the entire vulva. The earlier vulvar cancer is diagnosed, the less likely an extensive surgery is needed for treatment. Symptoms.	1.783429940541585
  what is fighting type pokemon weak against	Examples: 1 Electric-type attacks on a water/flying-type Pokemon mix for 4x damage. 2 This is because Electric-types are strong against both Water-and Flying-types. 3 Fighting-type attacks on a Flying/Poison-type Pokemon mix for .25x damage.	The second Zapdos, an Electric-type Electric Pokemon. Pokémon zapdos first appeared in the video game series which the player can Capture zapdos in The Power. Plant the last one Is, moltres A-fire Type Flame pokemon pokémon and one of the Three bird pokemon pokémon that is considered as. the most legendary	12.374764601389566

• Loss: MarginMSELoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity"
  }
  

Evaluation Dataset

msmarco

• Dataset: msmarco at 9e329ed
• Size: 10,000 evaluation samples
• Columns: query_id, positive_id, negative_id, and score
• Approximate statistics based on the first 1000 samples:

  	query_id	positive_id	negative_id	score
  type	string	string	string	float
  details	min: 11 charactersmean: 34.73 charactersmax: 158 characters	min: 76 charactersmean: 355.74 charactersmax: 1217 characters	min: 59 charactersmean: 339.1 charactersmax: 963 characters	min: -1.69mean: 13.45max: 22.14

• Samples:

  query_id	positive_id	negative_id	score
  does a cars registered owner have to be the insurance policy holder?	For example when a person has a company car they may be registered as the keeper though they are not the owner. Named driver-this is a driver who has been added to a car insurance policy to be an additional one to the main driver.	Features & Benefits Of Third Party Insurance Policy For Cars. 1 Death or bodily injury to a third party. 2 Damage to third party property. 3 Accidental death of the vehicle’s Owner or Driver. 4 Permanent Total Disability suffered by vehicle’s Owner or Driver.	5.590251604715983
  how to pronounce lucretia	Lucretia /lu-cre-tia/ [3 sylls.] as a girls' name is pronounced loo-KREE-shah. It is of Latin origin, and the meaning of Lucretia is succeed. Also possibly of Etruscan origin and of uncertain meaning. The name of a Roman matron who committed suicide in public protest against dishonor.	How To Pronounce Safat. English pronunciation for Safat is: Breaking a name down into syllables can make pronouncing it much easier. If you see the name Safat divided into smaller parts you can try to pronounce each part separately to get correct emphasis.	17.093592802683514
  average cost of gym equipment	When equipping a home-gym, it’s critical to consider at least one piece of cardio equipment. The top four treadmill picks from Consumer Search range from $990 to $3,500, with the average costing around $2,000.Another piece of popular cardio equipment is an elliptical trainer, which averages $1,500.he top four treadmill picks from Consumer Search range from $990 to $3,500, with the average costing around $2,000. Another piece of popular cardio equipment is an elliptical trainer, which averages $1,500.	“It’s what we all learned — or should have learned — in sixth-grade health class,” he said. “It’s all common sense. You need to keep yourself and your equipment clean. You never know who last used the equipment in a gym. It can be a great breeding ground for these bugs, some of which are pretty nasty.”	15.078913132349651

• Loss: MarginMSELoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Energy Consumed: 13.064 kWh
• Carbon Emitted: 4.822 kg of CO2
• Hours Used: 2.996 hours

Training Hardware

• On Cloud: No
• GPU Model: 8 x NVIDIA H100 80GB HBM3
• CPU Model: AMD EPYC 7R13 Processor
• RAM Size: 1999.99 GB

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.9.1+cu126
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}