CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 256 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['19 [SEP] average cost of dj for wedding', "The Knot found the average cost of a wedding nationwide is $35,329 not including the honeymoon. That's up $2,688 from last year and it's the highest average since The Knot started tracking average wedding costs 10 years ago. So what's behind the jump?"],
    ['15 [SEP] what is the origin name glaser mean', 'Paul Michael Glaser. Paul Michael Glaser (born March 25, 1943) is an American actor and director, perhaps best known for his role as Detective David Starsky on the 1970s television series Starsky and Hutch; he also appeared as Captain Jack Steeper on the 1999 to 2005 NBC series Third Watch.'],
    ['14 [SEP] what is the ego', 'The ego is  your consciousness of your own identity, or the conscious mind. Basically, your ego is related to your confidence level; if you have a big ego, your confidence is high, and visa versa. Additionally, having an ego can mean having an inflated feeling of pride or self-worth. The ego is your consciousness of your own identity, or the conscious mind. Basically, your ego is related to your confidence level; if you have a big ego, your confidence is high, and visa versa. Additionally, having an ego can mean having an inflated feeling of pride or self-worth.'],
    ['10 [SEP] what is the virginia mason emmi program', "Although Oklahoma State receiver James Washington is only 6 feet tall, quarterback Mason Rudolph doesn't see his size as a negative in today's NFL. Although Oklahoma State receiver James Washington is only 6 feet tall, quarterback Mason Rudolph doesn't see his size as a negative in today's NFL."],
    ['16 [SEP] is there an airport in atlantic city?', "The Absecon station is served by New Jersey Transit 's Atlantic City Line trains, with east-west service between 30th Street Station in Philadelphia and the Atlantic City Rail Terminal in Atlantic City."],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    '19 [SEP] average cost of dj for wedding',
    [
        "The Knot found the average cost of a wedding nationwide is $35,329 not including the honeymoon. That's up $2,688 from last year and it's the highest average since The Knot started tracking average wedding costs 10 years ago. So what's behind the jump?",
        'Paul Michael Glaser. Paul Michael Glaser (born March 25, 1943) is an American actor and director, perhaps best known for his role as Detective David Starsky on the 1970s television series Starsky and Hutch; he also appeared as Captain Jack Steeper on the 1999 to 2005 NBC series Third Watch.',
        'The ego is  your consciousness of your own identity, or the conscious mind. Basically, your ego is related to your confidence level; if you have a big ego, your confidence is high, and visa versa. Additionally, having an ego can mean having an inflated feeling of pride or self-worth. The ego is your consciousness of your own identity, or the conscious mind. Basically, your ego is related to your confidence level; if you have a big ego, your confidence is high, and visa versa. Additionally, having an ego can mean having an inflated feeling of pride or self-worth.',
        "Although Oklahoma State receiver James Washington is only 6 feet tall, quarterback Mason Rudolph doesn't see his size as a negative in today's NFL. Although Oklahoma State receiver James Washington is only 6 feet tall, quarterback Mason Rudolph doesn't see his size as a negative in today's NFL.",
        "The Absecon station is served by New Jersey Transit 's Atlantic City Line trains, with east-west service between 30th Street Station in Philadelphia and the Atlantic City Rail Terminal in Atlantic City.",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

Unnamed Dataset

• Size: 79,518,560 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 19 charactersmean: 42.8 charactersmax: 138 characters	min: 67 charactersmean: 346.56 charactersmax: 916 characters	min: -11.84mean: 0.64max: 11.02

• Samples:

  sentence_0	sentence_1	label
  19 [SEP] average cost of dj for wedding	The Knot found the average cost of a wedding nationwide is $35,329 not including the honeymoon. That's up $2,688 from last year and it's the highest average since The Knot started tracking average wedding costs 10 years ago. So what's behind the jump?	0.9657705624898275
  15 [SEP] what is the origin name glaser mean	Paul Michael Glaser. Paul Michael Glaser (born March 25, 1943) is an American actor and director, perhaps best known for his role as Detective David Starsky on the 1970s television series Starsky and Hutch; he also appeared as Captain Jack Steeper on the 1999 to 2005 NBC series Third Watch.	1.0410912434260051
  14 [SEP] what is the ego	The ego is your consciousness of your own identity, or the conscious mind. Basically, your ego is related to your confidence level; if you have a big ego, your confidence is high, and visa versa. Additionally, having an ego can mean having an inflated feeling of pride or self-worth. The ego is your consciousness of your own identity, or the conscious mind. Basically, your ego is related to your confidence level; if you have a big ego, your confidence is high, and visa versa. Additionally, having an ego can mean having an inflated feeling of pride or self-worth.	8.833635489145914

• Loss: FitMixinLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 1
• fp16: True

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs