SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("mazej/all-MiniLM-L6-v2_MultipleNegativesRankingLoss_b64_accs4_171531_fine-tuned")
# Run inference
sentences = [
    'Are files in a hidden directory safe?',
    "Are files in a hidden directory safe? Lets pretend that I have a website and want to keep personal notes on it. If I make a directory like mysite.com/84eW93e7 or mysite.com/34aWe897.txt where the letters and numbers are random so randomly guessing the file or directory name are impractical. How secure is this directory? Could someone still discover the existence of the file or directory assuming directory listing is off? If so, how would they find out? Of course this isnt something I would do, I'm just curious.",
    'I have used the ogr2ogr to import various formats to Postgres and have used GDAL utilities. Is there a complete Java port for these tools? Or are they simply wrappers calling JNI to C++ code? I have seen some Java stuff by googling but not sure how much are they current and kept upto date. Thanks, Ramesh',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: ir_eval
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 5,509 training samples
• Columns: query and document
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	min: 4 tokensmean: 13.07 tokensmax: 44 tokens	min: 12 tokensmean: 124.26 tokensmax: 256 tokens

• Samples:

  query	document
  Differnce between abilities of System apps, Apps in phone memory and Apps in the SD card	As I've played around with different ROMs and used Titanium Backup, I see that apps are categorized into system and user apps. Several ROM developers state that Titanium Backup should only be used for user apps and not for system apps, and when I proceed to uninstall system apps, TiBu warns me that the ROM may not work correctly. Other than the fact that user apps are downloadable via the Market or some other means, what are the key differences between the two? What is it about the system apps that make them more integral to the OS itself?
  Usage of Who and Whom	What's the rule for using who or whom? I was writing a LinkedIn recommendation one day, and ended up pondering for a while which of these forms to use: > … is a great developer whom I always found easy to work with. > > … is a great developer who I always found easy to work with. Both are basically correct in contemporary English, right? But is one or the other preferable, and if so, why? (In this case I went with the latter, as it seemed more common (Google) and I wanted to avoid sounding unnecessarily “archaic”, although I’m not sure whether that would have been the case. Also note that I try to write in a “friendly professional” style instead of overly formal one. :-)
  How do I override a \configure statement in html4.4t - tex4ht	Im trying to prepare html for conversion to an epub and have encountered a couple of problems that I would appreciate help with. Hope mixing a couple of related questions in a single post is OK. The easiest is the quotation environment doesnt indent both margins (and not enough on the left). Ive written a .cfg file and think I have that one solved. Heres what Im using: \Preamble{html} \begin{document} \Css { .quotation { margin-bottom:0.25em; margin-top:0.25em; margin-left:4em; margin-right:4em } } \EndPreamble The next is figures and tables. tex4ht inserts an hr tag before and after the table. This is not part of the css, its a separate tag generated by tex4ht. I havent found where one customizes this. I can delete these in emacs from the html, but that doesnt seem the right way to do this. Heres a fairly minimal example showing the rules (and the quotation indent as well). \documentclass{book} \begin{document} \chapter{Simple Example} Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer nec odio. Praesent libero. Sed cursus ante dapibus diam. Sed nisi. Nulla quis sem at nibh elementum imperdiet. Duis sagittis ipsum. Praesent mauris. Fusce nec tellus sed augue semper porta. Mauris massa. Vestibulum lacinia arcu eget nulla. \begin{quotation} Default book style icreases the left and right margins for quotations by a noticeable amount. The css generated by tex4ht only increases the left margin by one em. The attached .cfg increases this to 4 em. \end{quotation} Sed lectus. Integer euismod lacus luctus magna. Quisque cursus, metus vitae pharetra auctor, sem massa mattis sem, at interdum magna augue eget diam. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Morbi lacinia molestie dui. amet mauris. \begin{table} \begin{center} \hbox{ \begin{tabular}{

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• gradient_accumulation_steps: 4
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Framework Versions

• Python: 3.8.18
• Sentence Transformers: 3.2.1
• Transformers: 4.45.2
• PyTorch: 2.4.1
• Accelerate: 1.0.1
• Datasets: 3.1.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}