distilbert-base-uncased finetuned on MSMARCO with MarginMSELoss (BERT-CAT ensemble teacher)

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased on the msmarco dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 300 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Dot Product
• Supported Modality: Text
• Training Dataset:
  • msmarco
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kwondw/distilbert-base-uncased-msmarco-margin-mse-1000000")
# Run inference
queries = [
    'what is a dilated aorta',
]
documents = [
    'All about the Dilated Aorta. A dilated aorta is nothing to sneeze at; it is a serious condition that could result in death if left untreated. In fact, every year more than 10,000 Americans die from problems resulting from a dilated aorta.ymptoms of a Dilated Aorta. Although the vast majority of aneurysms donâ\x80\x99t present any symptoms until they rupture or lead to other severe problems, aneurysms can present certain signs. Abdominal aneurysms for example, will cause pain in the abdomen or numbness in the leg.',
    'Dilation is a medical term meaning that the internal diameter of something got larger. Like the pupils dilated or the cervix dilated. Mydriasis is dilation of the pupils. Ectasis is the medical term meaning dilation or expansion of a hollow organ. -ectasis is also the medical terminology combining form meaning dilation or expansion.',
    'Adj. 1. preceding-existing or coming before. antecedent-preceding in time or order. succeeding-coming after or following. previous-Preceding means occurring immediately before the time of the utterance, while previous means occurring at some time before the utterance..',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[88.0153, 81.6854, 63.3136]])

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact, NanoTouche2020, NanoMSMARCO, NanoNFCorpus and NanoNQ
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ],
      "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Training Details

Training Dataset

msmarco

• Dataset: msmarco
• Size: 1,000,000 training samples
• Columns: score, query, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	score	query	positive	negative
  type	float	string	string	string
  details	min: -1.45mean: 13.32max: 22.35	min: 4 tokensmean: 8.99 tokensmax: 22 tokens	min: 19 tokensmean: 81.77 tokensmax: 282 tokens	min: 20 tokensmean: 77.38 tokensmax: 233 tokens

• Samples:

  score	query	positive	negative
  18.26768692334493	weisskopf syndrome	Dinno-Shearer-Weisskopf syndrome: Introduction. Dinno-Shearer-Weisskopf syndrome: A very rare syndrome characterized mainly by long limbs, tall stature, large head, ataxia and facial anomalies. More detailed information about the symptoms, causes, and treatments of Dinno-Shearer-Weisskopf syndrome is available below.	Related posts: 1 Charles Bonnet Syndrome - Treatment, Symptoms and Causes. 2 Refeeding Syndrome - Symptoms, Causes, Diagnosis, Treatment Guidelines. 3 Dubowitz Syndrome.
  21.609440167744957	weather in beirut, lebanon	Beirut: Annual Weather Averages. August is the hottest month in Beirut with an average temperature of 28°C (82°F) and the coldest is February at 13°C (55°F) with the most daily sunshine hours at 14 in July. The wettest month is January with an average of 130mm of rain.	When Brant and his Mohawk warriors joined the fortunes. of the Johnson family in Canada, the influence of President. Whelock, of Lebanon Institute, where Brant had received his. early education, was brought to bear upon the crisis.
  15.881475607554119	what are the clots in my period	Blood Clots during Period. It is normal to pass blood clots during menses, as along with the blood the thickened lining of the uterus is also being shed and expelled outside the body. In case the abnormal blood clots during period occur, visit your gynecologist as it may then be an indication of other serious medical illnesses.	Heparin lock flush is used for: Preventing blood clots from forming in intravenous (IV) lines or catheters. Heparin lock flush is an anticoagulant. It works by preventing blood clots from forming in intravenous lines when they are not in use. Do NOT use heparin lock flush if:

• Loss: MarginMSELoss with these parameters:

  {
      "similarity_fct": "pairwise_dot_score"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 1
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 1.5 hours
• Evaluation: 2.2 minutes
• Total: 1.5 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}