SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'radcliffe eye hard line drug paula radcliffe called athlete found guilty drug charge treated criminal marathon world record holder belief need done rid athletics suspicion innuendo greet fast time doping sport criminal offence treated year old told sunday time cheat athlete also cheat promoter sponsor general public radcliffe comment come time several american sport star suspicion steroid use caught possession performance enhancing drug carry penalty added current system detect many substance abused athlete mean often athlete know competing level playing field hard work sacrifice trumped easier scientific route often athlete put good performance subjected suspicion innuendo instead praise receiving end accusation like testify much hurt',
    'radcliffe eye hard line drug paula radcliffe called athlete found guilty drug charge treated criminal marathon world record holder belief need done rid athletics suspicion innuendo greet fast time doping sport criminal offence treated year old told sunday time cheat athlete also cheat promoter sponsor general public radcliffe comment come time several american sport star suspicion steroid use caught possession performance enhancing drug carry penalty added current system detect many substance abused athlete mean often athlete know competing level playing field hard work sacrifice trumped easier scientific route often athlete put good performance subjected suspicion innuendo instead praise receiving end accusation like testify much hurt',
    'n extends indian beer venture uk biggest brewer scottish newcastle n buy india united brewery deal worth bn rupee n buy equity stake united maker well known kingfisher lager brand make public offer buy another stake similar holding controlled vijay mallya chair indian firm deal natural development joint venture united said tony froggatt n chief executive top brand include newcastle brown ale foster john smith strongbow kronenbourg n united agreed form strategic partnership one would include joint venture business uk investment indian brewer joint venture established may party stake venture millennium alcobev millennium alcobev merged united expects post merger half india beer market india population one billion consumes billion bottle beer every year kingfisher market share addition equity stake n invest bn rupee united non convertible redeemable preference share meanwhile united budget airline kingfisher airline buy aircraft airbus option buy aircraft deal worth bn airline brainchild mr mallya expects start operation end april new airline would break even first year operation mr mallya said',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  1.0000, -0.0705],
#         [ 1.0000,  1.0000, -0.0705],
#         [-0.0705, -0.0705,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,225 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 55 tokensmean: 204.0 tokensmax: 256 tokens	min: 55 tokensmean: 204.0 tokensmax: 256 tokens

• Samples:

  sentence_0	sentence_1
  tory opposing hour drinking tory say plan extend pub opening time put hold binge drinking control despite backing law change last year spokesman david davis said minister failed make party aware concern among senior police plan would cause anti social behaviour notts police chief steve green said innocent people would suffer culture secretary tessa jowell said delay would disastrous accused tory opportunism government would go ahead change would give police power tackle excessive drinking added earlier chief constable green questioned officer would able practically apply power allowing shut problem premise look market square nottingham fight take place licensed premise go lay responsibilty door asked bbc radio world one programme warned drinking establishment allowed open three four morning police would take officer day shift order job effectively night earlier year royal college physician said opposed plan extend drinking hour already epidemic binge drinking minister richard caborn sa...	tory opposing hour drinking tory say plan extend pub opening time put hold binge drinking control despite backing law change last year spokesman david davis said minister failed make party aware concern among senior police plan would cause anti social behaviour notts police chief steve green said innocent people would suffer culture secretary tessa jowell said delay would disastrous accused tory opportunism government would go ahead change would give police power tackle excessive drinking added earlier chief constable green questioned officer would able practically apply power allowing shut problem premise look market square nottingham fight take place licensed premise go lay responsibilty door asked bbc radio world one programme warned drinking establishment allowed open three four morning police would take officer day shift order job effectively night earlier year royal college physician said opposed plan extend drinking hour already epidemic binge drinking minister richard caborn sa...
  briton fed net service survey conducted pc pro magazine revealed many briton unhappy internet service fed slow speed high price level customer service receive reader switched supplier considering changing near future particularly bad news bt uk biggest internet supplier almost three time many people trying leave joining third broadband user interviewed fed current provider could tip iceberg think tim danton editor pc pro magazine expect figure leap every month price drop people trying switch said survey found bt tiscali actively dissuading customer leaving offering lower price phone cancel subscription reader offered price drop p expensive offered alternative operator making hardly worth swapping found tied month contract broadband become hugely competitive provider desperate hold customer surveyed found unable swap discovered huge variety problem one biggest issue current supplier withholding information people need give new supplier said tim danton editor pc pro break code practice c...	briton fed net service survey conducted pc pro magazine revealed many briton unhappy internet service fed slow speed high price level customer service receive reader switched supplier considering changing near future particularly bad news bt uk biggest internet supplier almost three time many people trying leave joining third broadband user interviewed fed current provider could tip iceberg think tim danton editor pc pro magazine expect figure leap every month price drop people trying switch said survey found bt tiscali actively dissuading customer leaving offering lower price phone cancel subscription reader offered price drop p expensive offered alternative operator making hardly worth swapping found tied month contract broadband become hugely competitive provider desperate hold customer surveyed found unable swap discovered huge variety problem one biggest issue current supplier withholding information people need give new supplier said tim danton editor pc pro break code practice c...
  joke e mail virus trick user virus disguise joke spreading rapidly across net anti virus firm issuing high level warning new version bagle e mail program seems catching lot people window virus grab e mail address microsoft outlook us mail sending software spread new victim infects machine bagle variant turn security measure usually protect pc new variant called bagle bagle bb bagle au attachment bearing virus code labelled either joke price body virus usually contains nothing smiley emoticon virus strike computer running window nt xp user infected open attachment travel e mail well plundering microsoft outlook e mail address send bagle also try turn firewall security centre service window xp machine bbc news online received five warning virus security company finnish company f secure gave virus second highest threat level several report world said mikko hypponen director anti virus research f secure security firm network box said stopped copy hour virus outbreak reached peak black spid...	joke e mail virus trick user virus disguise joke spreading rapidly across net anti virus firm issuing high level warning new version bagle e mail program seems catching lot people window virus grab e mail address microsoft outlook us mail sending software spread new victim infects machine bagle variant turn security measure usually protect pc new variant called bagle bagle bb bagle au attachment bearing virus code labelled either joke price body virus usually contains nothing smiley emoticon virus strike computer running window nt xp user infected open attachment travel e mail well plundering microsoft outlook e mail address send bagle also try turn firewall security centre service window xp machine bbc news online received five warning virus security company finnish company f secure gave virus second highest threat level several report world said mikko hypponen director anti virus research f secure security firm network box said stopped copy hour virus outbreak reached peak black spid...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Time

• Training: 21.0 seconds

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}