SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ayushexel/emb-all-MiniLM-L6-v2-squad-3-epochs")
# Run inference
sentences = [
    'In what region of modern-day France did the Vikings settle?',
    "The breakup of the Carolingian Empire was accompanied by invasions, migrations, and raids by external foes. The Atlantic and northern shores were harassed by the Vikings, who also raided the British Isles and settled there as well as in Iceland. In 911, the Viking chieftain Rollo (d. c. 931) received permission from the Frankish King Charles the Simple (r. 898–922) to settle in what became Normandy.[Q] The eastern parts of the Frankish kingdoms, especially Germany and Italy, were under continual Magyar assault until the invader's defeat at the Battle of Lechfeld in 955. The breakup of the Abbasid dynasty meant that the Islamic world fragmented into smaller political states, some of which began expanding into Italy and Sicily, as well as over the Pyrenees into the southern parts of the Frankish kingdoms.",
    "The breakup of the Carolingian Empire was accompanied by invasions, migrations, and raids by external foes. The Atlantic and northern shores were harassed by the Vikings, who also raided the British Isles and settled there as well as in Iceland. In 911, the Viking chieftain Rollo (d. c. 931) received permission from the Frankish King Charles the Simple (r. 898–922) to settle in what became Normandy.[Q] The eastern parts of the Frankish kingdoms, especially Germany and Italy, were under continual Magyar assault until the invader's defeat at the Battle of Lechfeld in 955. The breakup of the Abbasid dynasty meant that the Islamic world fragmented into smaller political states, some of which began expanding into Italy and Sicily, as well as over the Pyrenees into the southern parts of the Frankish kingdoms.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: gooqa-dev
• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 44,285 training samples
• Columns: question, context, and negative
• Approximate statistics based on the first 1000 samples:

  	question	context	negative
  type	string	string	string
  details	min: 7 tokensmean: 14.28 tokensmax: 36 tokens	min: 30 tokensmean: 147.82 tokensmax: 256 tokens	min: 31 tokensmean: 146.59 tokensmax: 256 tokens

• Samples:

  question	context	negative
  What location was originally featured in the Monopoly game?	Atlantic City (sometimes referred to as "Monopoly City") has become well-known over the years for its portrayal in the U.S. version of the popular board game, Monopoly, in which properties on the board are named after locations in and near Atlantic City. While the original incarnation of the game did not feature Atlantic City, it was in Indianapolis that Ruth Hoskins learned the game, and took it back to Atlantic City. After she arrived, Hoskins made a new board with Atlantic City street names, and taught it to a group of local Quakers.	As the Nintendo Entertainment System grew in popularity and entered millions of American homes, some small video rental shops began buying their own copies of NES games, and renting them out to customers for around the same price as a video cassette rental for a few days. Nintendo received no profit from the practice beyond the initial cost of their game, and unlike movie rentals, a newly released game could hit store shelves and be available for rent on the same day. Nintendo took steps to stop game rentals, but didn't take any formal legal action until Blockbuster Video began to make game rentals a large-scale service. Nintendo claimed that allowing customers to rent games would significantly hurt sales and drive up the cost of games. Nintendo lost the lawsuit, but did win on a claim of copyright infringement. Blockbuster was banned from including original, copyrighted instruction booklets with their rented games. In compliance with the ruling, Blockbuster produced their own short in...
  How many data transfer ports are included in most current Macs?	The current Mac product family uses Intel x86-64 processors. Apple introduced an emulator during the transition from PowerPC chips (called Rosetta), much as it did during the transition from Motorola 68000 architecture a decade earlier. The Macintosh is the only mainstream computer platform to have successfully transitioned to a new CPU architecture, and has done so twice. All current Mac models ship with at least 8 GB of RAM as standard other than the 1.4 GHz Mac Mini, MacBook Pro (without Retina Display), and MacBook Air. Current Mac computers use ATI Radeon or nVidia GeForce graphics cards as well as Intel graphics built into the main CPU. All current Macs (except for the MacBook Pro without Retina Display) do not ship with an optical media drive that includes a dual-function DVD/CD burner. Apple refers to this as a SuperDrive. Current Macs include two standard data transfer ports: USB and Thunderbolt (except for the MacBook (2015 version), which only has a USB-C port and headphone ...	The sales breakdown of the Macintosh have seen sales of desktop Macs stayed mostly constant while being surpassed by that of Mac notebooks whose sales rate has grown considerably; seven out of ten Macs sold were laptops in 2009, a ratio projected to rise to three out of four by 2010. The change in sales of form factors is due to the desktop iMac moving from affordable (iMac G3) to upscale (iMac G4) and subsequent releases are considered premium all-in-ones. By contrast the MSRP of the MacBook laptop lines have dropped through successive generations such that the MacBook Air and MacBook Pro constitute the lowest price of entry to a Mac, with the exception of the even more inexpensive Mac Mini (the only sub-$1000 offering from Apple, albeit without a monitor and keyboard), not surprisingly the MacBooks are the top-selling form factors of the Macintosh platform today. The use of Intel microprocessors has helped Macs more directly compete with their Windows counterparts on price and perfor...
  What sleep aid was marketed in 1904?	In 1903 Hermann Emil Fischer and Joseph von Mering disclosed their discovery that diethylbarbituric acid, formed from the reaction of diethylmalonic acid, phosphorus oxychloride and urea, induces sleep in dogs. The discovery was patented and licensed to Bayer pharmaceuticals, which marketed the compound under the trade name Veronal as a sleep aid beginning in 1904. Systematic investigations of the effect of structural changes on potency and duration of action led to the discovery of phenobarbital at Bayer in 1911 and the discovery of its potent anti-epileptic activity in 1912. Phenobarbital was among the most widely used drugs for the treatment of epilepsy through the 1970s, and as of 2014, remains on the World Health Organizations list of essential medications. The 1950s and 1960s saw increased awareness of the addictive properties and abuse potential of barbiturates and amphetamines and led to increasing restrictions on their use and growing government oversight of prescribers. Today...	Tito's estrangement from the USSR enabled Yugoslavia to obtain US aid via the Economic Cooperation Administration (ECA), the same US aid institution which administered the Marshall Plan. Still, he did not agree to align with the West, which was a common consequence of accepting American aid at the time. After Stalin's death in 1953, relations with the USSR were relaxed and he began to receive aid as well from the COMECON. In this way, Tito played East-West antagonism to his advantage. Instead of choosing sides, he was instrumental in kick-starting the Non-Aligned Movement, which would function as a 'third way' for countries interested in staying outside of the East-West divide.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: question, context, and negative_1
• Approximate statistics based on the first 1000 samples:

  	question	context	negative_1
  type	string	string	string
  details	min: 3 tokensmean: 14.53 tokensmax: 38 tokens	min: 33 tokensmean: 144.53 tokensmax: 256 tokens	min: 36 tokensmean: 146.07 tokensmax: 256 tokens

• Samples:

  question	context	negative_1
  What do some viruses store their genome in instead of DNA?	The concept of a gene continues to be refined as new phenomena are discovered. For example, regulatory regions of a gene can be far removed from its coding regions, and coding regions can be split into several exons. Some viruses store their genome in RNA instead of DNA and some gene products are functional non-coding RNAs. Therefore, a broad, modern working definition of a gene is any discrete locus of heritable, genomic sequence which affect an organism's traits by being expressed as a functional product or by regulation of gene expression.	In modern molecular biology and genetics, the genome is the genetic material of an organism. It consists of DNA (or RNA in RNA viruses). The genome includes both the genes and the non-coding sequences of the DNA/RNA.
  Who was the artist of the first song used?	The song "Tom's Diner" by Suzanne Vega was the first song used by Karlheinz Brandenburg to develop the MP3. Brandenburg adopted the song for testing purposes, listening to it again and again each time refining the scheme, making sure it did not adversely affect the subtlety of Vega's voice.	Influences also came to her from the art world, most notably through the works of Mexican artist Frida Kahlo. The music video of the song "Bedtime Story" featured images inspired by the paintings of Kahlo and Remedios Varo. Madonna is also a collector of Tamara de Lempicka's Art Deco paintings and has included them in her music videos and tours. Her video for "Hollywood" (2003) was an homage to the work of photographer Guy Bourdin; Bourdin's son subsequently filed a lawsuit for unauthorised use of his father's work. Pop artist Andy Warhol's use of sadomasochistic imagery in his underground films were reflected in the music videos for "Erotica" and "Deeper and Deeper".
  Who encouraged peanut production in Guinea-Bissau?	Rice is a staple in the diet of residents near the coast and millet a staple in the interior. Fish, shellfish, fruits and vegetables are commonly eaten along with cereal grains, milk, curd and whey. The Portuguese encouraged peanut production. Vigna subterranea (Bambara groundnut) and Macrotyloma geocarpum (Hausa groundnut) are also grown. Black-eyed peas are also part of the diet. Palm oil is harvested.	Guinea-Bissau was once part of the kingdom of Gabu, as well as part of the Mali Empire. Parts of this kingdom persisted until the 18th century, while a few others were under some rule by the Portuguese Empire since the 16th century. In the 19th century, it was colonized as Portuguese Guinea. Upon independence, declared in 1973 and recognised in 1974, the name of its capital, Bissau, was added to the country's name to prevent confusion with Guinea (formerly French Guinea). Guinea-Bissau has a history of political instability since independence, and no elected president has successfully served a full five-year term.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}