MiniLM-L12-H384 trained on GooAQ

This is a Cross Encoder model finetuned from microsoft/MiniLM-L12-H384-uncased on the gooaq dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: microsoft/MiniLM-L12-H384-uncased
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text
• Training Dataset:
  • gooaq
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("kwondw/reranker-MiniLM-L12-H384-uncased-gooaq-cmnrl")
# Get scores for pairs of inputs
pairs = [
    ['is a chloroplast in plant and animal cells?', "Both animal and plant cells have mitochondria, but only plant cells have chloroplasts. Plants don't get their sugar from eating food, so they need to make sugar from sunlight. This process (photosynthesis) takes place in the chloroplast."],
    ['game where the exorcist pops up?', "Once the player reaches the end of the third level in the game, a picture of Regan McNeil's face (the possessed girl from the 1973 horror movie The Exorcist) suddenly pops up on the screen, accompanied by a loud scream."],
    ['what is the difference between certified refurbished and manufacturer refurbished?', 'A certified refurbished will be fixed using new parts for parts that are worn or broken. ... A “just refurbished” item may just have the broken part repaired and it could be replaced with another used but good part or a part that may not meet the manufactures specs for quality.'],
    ['how much does it cost to replace the bearings in a hotpoint washing machine?', 'Expect to pay between $150 and $185 to replace drum bearings, including the cost of labor. Bearings last about 10 years. Call a washer repair contractor near you if it starts making unusual sounds. Broken bearings can lead to more issues and costly repairs if not fixed.'],
    ['select the correct answer. who coined the term jazz age?', 'F. Scott Fitzgerald is credited with coining the phrase “The Jazz Age” in the title of his 1922 collection of short stories, Tales of the Jazz Age.'],
]
scores = model.predict(pairs)
print(scores)
# [0.9585 0.9579 0.9585 0.9566 0.1592]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'is a chloroplast in plant and animal cells?',
    [
        "Both animal and plant cells have mitochondria, but only plant cells have chloroplasts. Plants don't get their sugar from eating food, so they need to make sugar from sunlight. This process (photosynthesis) takes place in the chloroplast.",
        "Once the player reaches the end of the third level in the game, a picture of Regan McNeil's face (the possessed girl from the 1973 horror movie The Exorcist) suddenly pops up on the screen, accompanied by a loud scream.",
        'A certified refurbished will be fixed using new parts for parts that are worn or broken. ... A “just refurbished” item may just have the broken part repaired and it could be replaced with another used but good part or a part that may not meet the manufactures specs for quality.',
        'Expect to pay between $150 and $185 to replace drum bearings, including the cost of labor. Bearings last about 10 years. Call a washer repair contractor near you if it starts making unusual sounds. Broken bearings can lead to more issues and costly repairs if not fixed.',
        'F. Scott Fitzgerald is credited with coining the phrase “The Jazz Age” in the title of his 1922 collection of short stories, Tales of the Jazz Age.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Reranking

• Datasets: NanoMSMARCO_R100, NanoNFCorpus_R100, NanoNQ_R100, NanoMSMARCO_R100, NanoNFCorpus_R100 and NanoNQ_R100
• Evaluated with CrossEncoderRerankingEvaluator with these parameters:

  {
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Cross Encoder Nano BEIR

• Dataset: NanoBEIR_R100_mean
• Evaluated with CrossEncoderNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en",
      "rerank_k": 100,
      "at_k": 10,
      "always_rerank_positives": true
  }
  

Training Details

Training Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 49,500 training samples
• Columns: question and answer
• Approximate statistics based on the first 100 samples:

  	question	answer
  type	string	string
  modality	text	text
  details	min: 8 tokensmean: 11.97 tokensmax: 19 tokens	min: 19 tokensmean: 62.12 tokensmax: 145 tokens

• Samples:

  question	answer
  what to spend my nectar points on?	There are lots of different places and ways you can spend your points. You can spend straight from your card by swiping it at Sainsbury's, Vue Cinemas and Argos or you can spend them online at nectar.com on great things like holidays, hotels and days out.
  what are the differences between bacteria archaea and eukarya?	There are three domains of life: Bacteria (also known as Eubacteria), Archaea, and Eukarya. The Bacteria and Archaea are made up entirely of microorganisms; the Eukarya contains plants, animals, and microorganisms such as fungi and protists. ... Here are other major differences between the three domains.
  why django is better than laravel?	Django takes security quite seriously and helps developers avoid the common mistakes of web development and implement some security best measures. While Laravel does also cover the basics of security it doesn't live up to Django's security level. That's why, for example, NASA uses Django for their website.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 10.0,
      "num_negatives": 5,
      "activation_fn": "torch.nn.modules.activation.Sigmoid",
      "mini_batch_size": 32
  }
  

Evaluation Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 500 evaluation samples
• Columns: question and answer
• Approximate statistics based on the first 100 samples:

  	question	answer
  type	string	string
  modality	text	text
  details	min: 8 tokensmean: 12.33 tokensmax: 22 tokens	min: 21 tokensmean: 60.41 tokensmax: 125 tokens

• Samples:

  question	answer
  is a chloroplast in plant and animal cells?	Both animal and plant cells have mitochondria, but only plant cells have chloroplasts. Plants don't get their sugar from eating food, so they need to make sugar from sunlight. This process (photosynthesis) takes place in the chloroplast.
  game where the exorcist pops up?	Once the player reaches the end of the third level in the game, a picture of Regan McNeil's face (the possessed girl from the 1973 horror movie The Exorcist) suddenly pops up on the screen, accompanied by a loud scream.
  what is the difference between certified refurbished and manufacturer refurbished?	A certified refurbished will be fixed using new parts for parts that are worn or broken. ... A “just refurbished” item may just have the broken part repaired and it could be replaced with another used but good part or a part that may not meet the manufactures specs for quality.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 10.0,
      "num_negatives": 5,
      "activation_fn": "torch.nn.modules.activation.Sigmoid",
      "mini_batch_size": 32
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• bf16: True
• per_device_eval_batch_size: 64
• seed: 12

All Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 64
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 1.7 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.0
• Transformers: 5.13.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Reranker Models with Sentence Transformers: the end-to-end guide for training or finetuning Cross Encoder (reranker) models.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video reranker models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: training multimodal Cross Encoders.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}