CrossEncoder based on microsoft/MiniLM-L12-H384-uncased

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased on the ms_marco dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text
• Training Dataset:
  • ms_marco
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of inputs
pairs = [
    ['meaning of name sebastian', "The name Sebastian is a Greek baby name. In Greek the meaning of the name Sebastian is: Revered. American Meaning: The name Sebastian is an American baby name. In American the meaning of the name Sebastian is: Revered. Latin Meaning: The name Sebastian is a Latin baby name. In Latin the meaning of the name Sebastian is: Venerable; revered. The twin brother of Viola in Shakespeare's Twelfth Night. A third-century martyred centurion who became St. Sebastian patron saint of soldiers."],
    ['meaning of name sebastian', 'It is of Greek origin, and the meaning of Sebastian is revered. The original form of this name referred to those from a particular city or region of Asia minor, whose Greek name was from the Latin imperial title Augustus.'],
    ['meaning of name sebastian', 'Sebastian is a widely used name; it has 39 variant forms that are used in English and other languages. English variant forms of Sebastian include Bastiano, Bastion, Seba, Sebastain, Sebasten, Sebastiane, Sebastin, Sebastine, Sebastion, Sebastyen, Sebastyn, Sebby, and Steb.'],
    ['meaning of name sebastian', 'Notes: Sebastian is a name with Greek origins. The meaning of the name Sebastian is venerable. Alternate spellings include Sabastian and Sebastion. Common nicknames include Seb, Bas, Bash, and Subby. The popularity of the name increased with the telling of the Biblical story of Saint Sebastian.'],
    ['meaning of name sebastian', 'From the Latin name Sebastianus which meant from Sebaste. Sebaste was the name a town in Asia Minor, its name deriving from Greek σεβαστος (sebastos) venerable (a translation of Latin Augustus, the title of the Roman emperors).'],
]
scores = model.predict(pairs)
print(scores)
# [0.9124 0.3346 0.3149 0.6478 0.3775]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'meaning of name sebastian',
    [
        "The name Sebastian is a Greek baby name. In Greek the meaning of the name Sebastian is: Revered. American Meaning: The name Sebastian is an American baby name. In American the meaning of the name Sebastian is: Revered. Latin Meaning: The name Sebastian is a Latin baby name. In Latin the meaning of the name Sebastian is: Venerable; revered. The twin brother of Viola in Shakespeare's Twelfth Night. A third-century martyred centurion who became St. Sebastian patron saint of soldiers.",
        'It is of Greek origin, and the meaning of Sebastian is revered. The original form of this name referred to those from a particular city or region of Asia minor, whose Greek name was from the Latin imperial title Augustus.',
        'Sebastian is a widely used name; it has 39 variant forms that are used in English and other languages. English variant forms of Sebastian include Bastiano, Bastion, Seba, Sebastain, Sebasten, Sebastiane, Sebastin, Sebastine, Sebastion, Sebastyen, Sebastyn, Sebby, and Steb.',
        'Notes: Sebastian is a name with Greek origins. The meaning of the name Sebastian is venerable. Alternate spellings include Sabastian and Sebastion. Common nicknames include Seb, Bas, Bash, and Subby. The popularity of the name increased with the telling of the Biblical story of Saint Sebastian.',
        'From the Latin name Sebastianus which meant from Sebaste. Sebaste was the name a town in Asia Minor, its name deriving from Greek σεβαστος (sebastos) venerable (a translation of Latin Augustus, the title of the Roman emperors).',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Training Details

Training Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 78,704 training samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 100 samples:

  	query	docs	labels
  type	string	list	list
  modality	text
  details	min: 4 tokensmean: 9.38 tokensmax: 17 tokens	min: 4 elementsmean: 8.24 elementsmax: 10 elements	min: 4 elementsmean: 8.24 elementsmax: 10 elements

• Samples:

  query	docs	labels
  what is protein shakes for	["Protein shakes are used mainly by athletes who need nourishment right after their workouts, says Jose Antonio, chief executive officer and co-founder of the International Society of Sports Nutrition (ISSN). Protein shakes can range from 100% protein to mostly carbohydrates with a little added protein and fat. Protein shakes vary in protein content. If you're a body builder, you're going to shift to the drinks that have a bit more protein, Antonio says. If you're an endurance athlete, like a marathoner, you may favor drinks with more carbs", 'What Type to Use? There are four main types of protein in powders and shakes – whey, casein, soy and/or rice. Whey and casein are both derived from milk (the protein in milk is 80% casein and 20% whey). Most protein products are made with whey, which is a “complete” protein and contains the highest branched chain amino acid (BCAA) content found in nature.', "The shakes tend to consist of whey protein, although casein (milk), soy, egg, hemp, rice, and pea protein powders are also available. A buyer's guide to protein shakes. What is a protein shake. There are still many misconceptions regarding protein shakes. Whey is the most common base for the protein powder, as it contains all of the nine essential amino acids that facilitate the healing of damaged muscles. There are three main types of whey protein: isolate, hydrolysate, and concentrate.", "Although it's best to get protein through your diet, supplementing it with a combination of whey and casein is a good choice, as long as you tolerate dairy well, because both come from milk. Soy protein is another option. It's a plant-based protein. Protein shakes vary in protein content. If you're a body builder, you're going to shift to the drinks that have a bit more protein, Antonio says. If you're an endurance athlete, like a marathoner, you may favor drinks with more carbs", "Since protein contains calories, consuming too much can actually make losing weight more difficult — especially if you drink protein shakes in addition to your usual diet, and you're not exercising. The average adult needs 46 to 56 grams of protein a day, depending on weight and overall health. As long as you're eating a healthy diet, adding extra protein — either through protein shakes or other sources — isn't necessary. Remember, the key to losing weight is burning more calories than you consume.", ...]	[1, 0, 0, 0, 0, ...]
  nature area definition	["Definition of NATURAL AREA. : a geographical area (as in a city) having a physical and cultural individuality developed through natural growth rather than design or planning. ADVERTISEMENT. This word doesn't usually appear in our free dictionary, but the definition from our premium Unabridged Dictionary is offered here on a limited basis", 'Full Definition of NATURAL. 1. 2. a: being in accordance with or determined by nature b: having or constituting a classification based on features existing in nature. 3. a (1) : begotten as distinguished from adopted; also: legitimate. 4. 5. 6. : of or relating to nature as an object of study and research. ', 'Wilderness or wildland is a natural environment on Earth that has not been significantly modified by civilized human activity. The word wilderness derives from the notion of wildness—in other words, that which is not controlled by humans. The mere presence or activity of people does not disqualify an area from being wilderness..', 'Identifying and helping landowners protect their natural areas is one of the jobs of the Louisiana Natural Heritage Program. However, we have determined that a natural area does not mean the same thing to everyone, whether they are a forester, wildlife biologist, or a private landowner. Therefore, we have attempted to describe below what a natural area is as accepted by the LNHP. Restoration/Maintenance of the system is really what natural area management is about. Implementing Natural Area management… Some ways to implement management of your natural area site are: 1 Maintain/restore historical natural community composition and structure.', 'is an umbrella term used to describe a wide range of theories and ideals. One such theory is that nature encapsulates the entirety of the physical realm and reality. Another definition for nature is simply the phenomena which are observed in the natural world such as-plants, animals and other physical features. NATURE: Nature is simply the phenomena which are observed in the natural world such as-plants, animals and other physical features.. No related posts', ...]	[1, 0, 0, 0, 0, ...]
  what does erikson mean by gerotranscendence	["Erikson's stages of psychosocial development. Erikson's stages of psychosocial development, as articulated by Erik Erikson, is a psychoanalytic theory which identifies eight stages through which a healthily developing human should pass from infancy to late adulthood. However, mastery of a stage is not required to advance to the next stage. Erikson's stage theory characterizes an individual advancing through the eight life stages as a function of negotiating his or her biological forces and sociocultural forces. Each stage is characterized by a psychosocial crisis of these two conflicting forces (as shown in the table below)", '6 I. INTRODUCTION Many theories on life span identity and personality development (e.g., Erikson, 1982, Levinson, 1978) emphasise the importance of attaining a sense of integrity and serenity as well as a sense of life satisfaction at the end of one’s life. 21 One could argue that E. Erikson considers the development of integrity and wisdom to be the result of a lifelong development towards coping with maturation because wisdom, as the highest stage of development, gives the strength to be able to face the losses in old age and fear of death.', 'Associated with the later psychosocial stages envisaged by Erikson and Tornstam’s concept of gerotranscendence is the broader term, ‘wisdom’. Wisdom is generally regarded as being deposited in societal laws and traditions and accrued by individuals through life experience. First, Tornstam regards gerotranscendence as a radical reformulation of the disengagement theory promulgated in the early 1960s by Elaine Cumming and colleagues (Cumming, 1963, Cumming and Henry, 1961 and Cumming and Newell, 1960).', 'This is true to a degree although others (Joan Erikson in Erikson and Erikson, 1997 and Johnson and Barer, 1993) see Tornstam’s theory as being more concerned with liberation in the present. It has to be noted that the suggestion of Erikson et al. First, Tornstam regards gerotranscendence as a radical reformulation of the disengagement theory promulgated in the early 1960s by Elaine Cumming and colleagues (Cumming, 1963, Cumming and Henry, 1961 and Cumming and Newell, 1960).', 'The theory of gerotranscendence is based not only on a given research paradigm but also on a paradigm about the individual. Tornstam does not state explicitly how he regards the individual, but it appears that he considers him to be a biological and social being. The concept of gerotranscendence is based primarily on logic, since it is developed on the basis of a theory. The concept has been developed within the gerontological theory tradition, and Tornstam states that the disengagement theory has been very important to him.', ...]	[1, 0, 0, 0, 0, ...]

• Loss: PListMLELoss with these parameters:

  {
      "lambda_weight": "sentence_transformers.cross_encoder.losses.plist_mle.PListMLELambdaWeight",
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16,
      "respect_input_order": true
  }
  

Evaluation Dataset

ms_marco

• Dataset: ms_marco at a47ee7a
• Size: 1,000 evaluation samples
• Columns: query, docs, and labels
• Approximate statistics based on the first 100 samples:

  	query	docs	labels
  type	string	list	list
  modality	text
  details	min: 5 tokensmean: 9.14 tokensmax: 17 tokens	min: 4 elementsmean: 8.47 elementsmax: 10 elements	min: 4 elementsmean: 8.47 elementsmax: 10 elements

• Samples:

  query	docs	labels
  meaning of name sebastian	["The name Sebastian is a Greek baby name. In Greek the meaning of the name Sebastian is: Revered. American Meaning: The name Sebastian is an American baby name. In American the meaning of the name Sebastian is: Revered. Latin Meaning: The name Sebastian is a Latin baby name. In Latin the meaning of the name Sebastian is: Venerable; revered. The twin brother of Viola in Shakespeare's Twelfth Night. A third-century martyred centurion who became St. Sebastian patron saint of soldiers.", 'It is of Greek origin, and the meaning of Sebastian is revered. The original form of this name referred to those from a particular city or region of Asia minor, whose Greek name was from the Latin imperial title Augustus.', 'Sebastian is a widely used name; it has 39 variant forms that are used in English and other languages. English variant forms of Sebastian include Bastiano, Bastion, Seba, Sebastain, Sebasten, Sebastiane, Sebastin, Sebastine, Sebastion, Sebastyen, Sebastyn, Sebby, and Steb.', 'Notes: Sebastian is a name with Greek origins. The meaning of the name Sebastian is venerable. Alternate spellings include Sabastian and Sebastion. Common nicknames include Seb, Bas, Bash, and Subby. The popularity of the name increased with the telling of the Biblical story of Saint Sebastian.', 'From the Latin name Sebastianus which meant from Sebaste. Sebaste was the name a town in Asia Minor, its name deriving from Greek σεβαστος (sebastos) venerable (a translation of Latin Augustus, the title of the Roman emperors).']	[1, 0, 0, 0, 0]
  where was william shatner born	['Actor, director, author, singer William Shatner was born on March 22, 1931, in Montreal, Canada. Best known for his roles on Boston Legal and Star Trek, Shatner is one of the most recognizable stars working today. ', 'William Shatner was born in Cote Côte-Saint luc In montreal Montréal on, march 22 1931 to Joseph (And) Anna. Garmaise shatner he is A famous, canadian, actor, musician, singer author, film, director spokesman, and comedian best known for his Role As Captain. James t kirk in The Original star trek series and the. subsequent movies William Shatner was born in Cote Côte-Saint luc In montreal Montréal on, march 22 1931 to Joseph (And) Anna. garmaise shatner', 'Early life and education Edit. Shatner was born in the Cote Côte-Saint luc neighborhood Of, Montreal, Quebec, canada the son Of (anne Nee) née Garmaise And, joseph shatner a. Clothing manufacturer he has, Two sisters Joy. And farla his, Paternal Grandfather, wolf schattner anglicized the family Name. to shatner 177 pages on this wiki. William Shatner (born March 22, 1931) appears as attorney Denny Crane, one-third of the Boston area-based law firm Crane, Poole & Schmidt on Boston Legal.', "William Bill Shatner (born March 22, 1931) is a Canadian actor, singer, author, producer, director, spokesman, and comedian. He gained worldwide fame and became a cultural icon for his portrayal of James T. Kirk, Captain of the USS Enterprise, in the Star Trek franchise. Shatner's wife Gloria Rand divorced him in March 1969. After Star Trek was cancelled that year, Shatner experienced difficulty in finding work in the early 1970s having been somewhat typecast from his role as Kirk.", 'William Shatner has notched up an impressive 50-plus years in front of the camera, most recently displaying comedic talent, and being instantly recognizable to several generations of cult television fans as the square-jawed Captain James T. Kirk, commander of the starship U.S.S. Enterprise. After Star Trek folded, Shatner spent the rest of the decade and the 1970s making the rounds guest-starring on many prime-time television series, including Hawaii Five-O (1968), Marcus Welby, M.D. (1969) and Ironside (1967).', ...]	[1, 0, 0, 0, 0, ...]
  what does charismatic mean in the greek	["The word charismatic is derived from the Greek word charisma (meaning a grace or a gift) which is the term used in the Bible to describe a wide range of supernatural experiences (especially in 1 Corinthians 12-14). The first is the belief that the charismatic gifts of the Holy Spirit, such as tongues, prophecy, and miraculous healing, are still in effect today. Charismatics also hold that sometimes Bible passages are misinterpreted in order to declare that charismatic gifts are not valid for today's believers. [2] [3].", '1. Of, relating to, or characterized by charisma: the warmth of a naturally charismatic leader (Joyce Carol Oates). 2. Of, relating to, or being a type of Christianity that emphasizes personal religious experience and divinely inspired powers, as of healing, prophecy, and the gift of tongues.', 'Charisma is a certain quality of an individual personality by virtue of which he is set apart from ordinary men and treated as endowed with supernatural, superhuman, or at least specifically exceptional powers or qualities. These dialectic meanings influenced notable changes in Pentecostalism in the late 19th century, and charismatic movements in some mainline churches in the mid-20th century. The discussion in the 21st Century Religion section explores what charisma means in these and other religious groups.', 'The term charismatic derives from the Greek word χάρισμα charisma (gift, itself derived from χάρις, grace or favor), which means charism in English. ', 'Charismatic is an English version of the Greek charismata which basically means “grace gift.” The gifts which are typically associated with such are those which are viewed as being more supernatural in nature, i.e. prophecy, healing, and tongues. ', ...]	[1, 0, 0, 0, 0, ...]

• Loss: PListMLELoss with these parameters:

  {
      "lambda_weight": "sentence_transformers.cross_encoder.losses.plist_mle.PListMLELambdaWeight",
      "activation_fn": "torch.nn.modules.linear.Identity",
      "mini_batch_size": 16,
      "respect_input_order": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• bf16: True
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• seed: 12

All Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 41.6 minutes
• Evaluation: 7.1 minutes
• Total: 48.7 minutes

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.10.2
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

PListMLELoss

@inproceedings{lan2014position,
  title={Position-Aware ListMLE: A Sequential Learning Process for Ranking},
  author={Lan, Yanyan and Zhu, Yadong and Guo, Jiafeng and Niu, Shuzi and Cheng, Xueqi},
  booktitle={UAI},
  volume={14},
  pages={449--458},
  year={2014}
}