SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '[+30d w5h19] Banana, Ezekiel 4:9 Bread Organic Sprouted Whole Grain, Grade A Cage Free Omega-3 Enriched Medium Eggs, Sweet Potatoes; [+10d w1h18] Cookie Dough Ice Cream Chocolate Chip, Peanut Butter Ice Cream Cup, Blackberry Cucumber Sparkling Water, Kiwi Sandia Sparkling Water, Protein Power Cold Pressed Fruit Smoothie, Defense Up Cold Pressed Fruit Smoothie; [+30d w4h12] Blackberry Cucumber Sparkling Water, Peanut Butter Ice Cream Cup, Cookie Dough Ice Cream Chocolate Chip, Cran Raspberry Sparkling Water, Lemon Italian Sparkling Mineral Water, Natural Chicken & Sage Breakfast Sausage, Organic Jasmine Rice, Organic Riced Cauliflower, Country Cheddar Bowl; [+5d w2h10] Original Rotisserie Chicken. Next: +16d w0h14',
    'Product: Large Lemon. Aisle: fresh fruits. Department: produce.',
    'Product: Blood Orange Italian Soda. Aisle: soft drinks. Department: beverages.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3506, 0.3826],
#         [0.3506, 1.0000, 0.6395],
#         [0.3826, 0.6395, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: order-recommendation
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,246,220 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 76 tokensmean: 151.76 tokensmax: 208 tokens	min: 15 tokensmean: 20.12 tokensmax: 38 tokens

• Samples:

  anchor	positive
  [w1h14] Tuna Ventresca, in Olive Oil, Lightly Smoked Sardines in Olive Oil, Naturally Smoked Oysters in Pure Olive Oil, Bulgarian Yogurt, Organic Whole String Cheese, Organic 4% Milk Fat Whole Milk Cottage Cheese, Pure Sparkling Water, Organic Small Bunch Celery; [+7d w1h10] Bulgarian Yogurt, Organic 4% Milk Fat Whole Milk Cottage Cheese, Pure Sparkling Water, Organic Small Bunch Celery, Organic Whole String Cheese, Banana, Plus Cranberry Almond + Antioxidants with Macadamia Nuts Bar, Dark Chocolate Cinnamon Pecan Bar; [+15d w2h21] Pure Sparkling Water, Dark Chocolate Cinnamon Pecan Bar, Sparkling Water Grapefruit, Naturally Smoked Oysters in Pure Olive Oil. Next: +9d w4h10	Product: Bulgarian Yogurt. Aisle: yogurt. Department: dairy eggs.
  [w1h14] Tuna Ventresca, in Olive Oil, Lightly Smoked Sardines in Olive Oil, Naturally Smoked Oysters in Pure Olive Oil, Bulgarian Yogurt, Organic Whole String Cheese, Organic 4% Milk Fat Whole Milk Cottage Cheese, Pure Sparkling Water, Organic Small Bunch Celery; [+7d w1h10] Bulgarian Yogurt, Organic 4% Milk Fat Whole Milk Cottage Cheese, Pure Sparkling Water, Organic Small Bunch Celery, Organic Whole String Cheese, Banana, Plus Cranberry Almond + Antioxidants with Macadamia Nuts Bar, Dark Chocolate Cinnamon Pecan Bar; [+15d w2h21] Pure Sparkling Water, Dark Chocolate Cinnamon Pecan Bar, Sparkling Water Grapefruit, Naturally Smoked Oysters in Pure Olive Oil. Next: +9d w4h10	Product: Organic 4% Milk Fat Whole Milk Cottage Cheese. Aisle: other creams cheeses. Department: dairy eggs.
  [w1h14] Tuna Ventresca, in Olive Oil, Lightly Smoked Sardines in Olive Oil, Naturally Smoked Oysters in Pure Olive Oil, Bulgarian Yogurt, Organic Whole String Cheese, Organic 4% Milk Fat Whole Milk Cottage Cheese, Pure Sparkling Water, Organic Small Bunch Celery; [+7d w1h10] Bulgarian Yogurt, Organic 4% Milk Fat Whole Milk Cottage Cheese, Pure Sparkling Water, Organic Small Bunch Celery, Organic Whole String Cheese, Banana, Plus Cranberry Almond + Antioxidants with Macadamia Nuts Bar, Dark Chocolate Cinnamon Pecan Bar; [+15d w2h21] Pure Sparkling Water, Dark Chocolate Cinnamon Pecan Bar, Sparkling Water Grapefruit, Naturally Smoked Oysters in Pure Olive Oil. Next: +9d w4h10	Product: Organic Celery Hearts. Aisle: fresh vegetables. Department: produce.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 30.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 138,397 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 66 tokensmean: 150.77 tokensmax: 203 tokens	min: 14 tokensmean: 20.47 tokensmax: 44 tokens

• Samples:

  anchor	positive
  [w1h21] Large Alfresco Eggs, Organic Whole Milk, Organic Cream Cheese Bar, Organic Grape Tomatoes, Organic Raspberries, Organic Strawberries, Organic D'Anjou Pears, Organic Green Seedless Grapes, Small Hass Avocado, Organic Unsalted Diced Tomatoes, Pasta Sauce, Aged White Cheddar Gluten-Free Baked Rice And Corn Puffs, Crescent Rolls, Plain Greek Yogurt, Natural Free & Clear Dish Liquid, Organic Honey Sweet Whole Wheat Bread, Flaky Biscuits; [+13d w0h15] Strawberry Preserves, Organic Raspberries, Organic Frozen Peas. Next: +24d w0h21	Product: Organic Grape Tomatoes. Aisle: packaged vegetables fruits. Department: produce.
  [w1h21] Large Alfresco Eggs, Organic Whole Milk, Organic Cream Cheese Bar, Organic Grape Tomatoes, Organic Raspberries, Organic Strawberries, Organic D'Anjou Pears, Organic Green Seedless Grapes, Small Hass Avocado, Organic Unsalted Diced Tomatoes, Pasta Sauce, Aged White Cheddar Gluten-Free Baked Rice And Corn Puffs, Crescent Rolls, Plain Greek Yogurt, Natural Free & Clear Dish Liquid, Organic Honey Sweet Whole Wheat Bread, Flaky Biscuits; [+13d w0h15] Strawberry Preserves, Organic Raspberries, Organic Frozen Peas. Next: +24d w0h21	Product: Organic Strawberries. Aisle: fresh fruits. Department: produce.
  [w1h21] Large Alfresco Eggs, Organic Whole Milk, Organic Cream Cheese Bar, Organic Grape Tomatoes, Organic Raspberries, Organic Strawberries, Organic D'Anjou Pears, Organic Green Seedless Grapes, Small Hass Avocado, Organic Unsalted Diced Tomatoes, Pasta Sauce, Aged White Cheddar Gluten-Free Baked Rice And Corn Puffs, Crescent Rolls, Plain Greek Yogurt, Natural Free & Clear Dish Liquid, Organic Honey Sweet Whole Wheat Bread, Flaky Biscuits; [+13d w0h15] Strawberry Preserves, Organic Raspberries, Organic Frozen Peas. Next: +24d w0h21	Product: Organic Honey Sweet Whole Wheat Bread. Aisle: bread. Department: bakery.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 30.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 0.0001
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_steps: 9736
• dataloader_drop_last: True
• load_best_model_at_end: True
• dataloader_pin_memory: False
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 9736
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: False
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.2
• Transformers: 5.1.0
• PyTorch: 2.10.0
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}