CrossEncoder based on cross-encoder/ettin-reranker-68m-v1

This is a Cross Encoder model finetuned from cross-encoder/ettin-reranker-68m-v1 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: cross-encoder/ettin-reranker-68m-v1
• Maximum Sequence Length: 128 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 512, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Dense({'in_features': 512, 'out_features': 512, 'bias': False, 'activation_function': 'torch.nn.modules.activation.GELU', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): LayerNorm({'dimension': 512})
  (4): Dense({'in_features': 512, 'out_features': 1, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'scores'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of inputs
pairs = [
    ['Plan, organize, or maintain dental health programs.', 'Turn or reposition bedridden patients.'],
    ['Maintain required records of work hours, budgets, payrolls, and other information.', 'Process payroll information.'],
    ['Conduct research to develop or test medications, treatments, or procedures to prevent or control disease or injury.', 'Conduct clinical or basic research.'],
    ['Prepare and submit reports describing the results of security fixes.', 'Receive patients, schedule appointments, and maintain patient records.'],
    ['Determine work assignments and procedures.', 'Conduct inventories to maintain stock of clinical supplies.'],
]
scores = model.predict(pairs)
print(scores)
# [-2.6881  1.3212 -0.5072 -2.6118 -2.2695]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Plan, organize, or maintain dental health programs.',
    [
        'Turn or reposition bedridden patients.',
        'Process payroll information.',
        'Conduct clinical or basic research.',
        'Receive patients, schedule appointments, and maintain patient records.',
        'Conduct inventories to maintain stock of clinical supplies.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Correlation

• Dataset: val_correlation
• Evaluated with CrossEncoderCorrelationEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 295,091 training samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 100 samples:

  	sentence1	sentence2	label
  type	string	string	float
  modality	text	text
  details	min: 5 tokensmean: 17.71 tokensmax: 46 tokens	min: 8 tokensmean: 17.44 tokensmax: 40 tokens	min: 0.0mean: 0.28max: 0.78

• Samples:

  sentence1	sentence2	label
  Monitor customer preferences to determine focus of sales efforts.	Advise food service managers and organizations on sanitation, safety procedures, menu development, budgeting, and planning to assist with establishment, operation, and evaluation of food service facilities and nutrition programs.	0.07737485777805486
  Collaborate with colleagues to address teaching and research issues.	Collaborate with colleagues to address teaching and research issues.	0.6937963999029872
  Keep abreast of developments in the field by reading current literature, talking with colleagues, and participating in professional conferences.	Keep abreast of developments in the field by reading current literature, talking with colleagues, and participating in professional conferences.	0.6937963999029872

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 32,788 evaluation samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 100 samples:

  	sentence1	sentence2	label
  type	string	string	float
  modality	text	text
  details	min: 6 tokensmean: 17.59 tokensmax: 45 tokens	min: 6 tokensmean: 18.17 tokensmax: 55 tokens	min: 0.0mean: 0.28max: 0.82

• Samples:

  sentence1	sentence2	label
  Plan, organize, or maintain dental health programs.	Turn or reposition bedridden patients.	0.056361597787474016
  Maintain required records of work hours, budgets, payrolls, and other information.	Process payroll information.	0.8059258707828006
  Conduct research to develop or test medications, treatments, or procedures to prevent or control disease or injury.	Conduct clinical or basic research.	0.37034095223637237

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• learning_rate: 2e-05
• warmup_steps: 0.1
• per_device_eval_batch_size: 128
• load_best_model_at_end: True

All Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 128
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.4 hours

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.6.0
• Transformers: 5.16.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.23.2

Additional Resources

• Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}