SentenceTransformer based on BAAI/bge-large-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-large-en-v1.5. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-large-en-v1.5
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Represent this sentence for searching relevant passages: What issues exist with Intel Killer Control Center software?',
    'Improper access control for the Intel(R) Killer(TM) Control Center software before version 2.4.3337.0 may allow an authorized user to potentially enable escalation of privilege via local access.',
    'A content spoofing vulnerability in the following components allows to render html pages containing arbitrary plain text content, which might fool an end user: UI add-on for SAP NetWeaver (UI_Infra, 1.0), SAP UI Implementation for Decoupled Innovations (UI_700, 2.0): SAP NetWeaver 7.00 Implementation, SAP User Interface Technology (SAP_UI 7.4, 7.5, 7.51, 7.52). There is little impact as it is not possible to embed active contents such as JavaScript or hyperlinks.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.7885, -0.0462],
#         [ 0.7885,  1.0000, -0.1053],
#         [-0.0462, -0.1053,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 861,607 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 15 tokensmean: 19.66 tokensmax: 31 tokens	min: 24 tokensmean: 84.49 tokensmax: 256 tokens

• Samples:

  anchor	positive
  Represent this sentence for searching relevant passages: fuse filesystem sync issues	In the Linux kernel, the following vulnerability has been resolved:writeback: don't block sync for filesystems with no data integrity guaranteesAdd a SB_I_NO_DATA_INTEGRITY superblock flag for filesystems that cannotguarantee data persistence on sync (eg fuse). For superblocks with thisflag set, sync kicks off writeback of dirty inodes but does not waitfor the flusher threads to complete the writeback.This replaces the per-inode AS_NO_DATA_INTEGRITY mapping flag added incommit f9a49aa302a0 ("fs/writeback: skip AS_NO_DATA_INTEGRITY mappingsin wait_sb_inodes()"). The flag belongs at the superblock level becausedata integrity is a filesystem-wide property, not a per-inode one.Having this flag at the superblock level also allows us to skip havingto iterate every dirty inode in wait_sb_inodes() only to skip each inodeindividually.Prior to this commit, mappings with no data integrity guarantees skippedwaiting on writeback completion but still waited on the flusher threadst...
  Represent this sentence for searching relevant passages: EMC Documentum WebTop CSRF issue	Cross-site request forgery (CSRF) vulnerability in EMC Documentum WebTop before 6.8P01, Documentum Administrator through 7.2, Documentum Digital Assets Manager through 6.5SP6, Documentum Web Publishers through 6.5SP7, and Documentum Task Space through 6.7SP2 allows remote attackers to hijack the authentication of arbitrary users. NOTE: this vulnerability exists because of an incomplete fix for CVE-2014-2518.
  Represent this sentence for searching relevant passages: sensitive data exposed on Windows 10	An information disclosure vulnerability exists when the Windows kernel improperly handles objects in memory, aka "Windows Kernel Information Disclosure Vulnerability." This affects Windows 7, Windows Server 2012 R2, Windows RT 8.1, Windows Server 2008, Windows Server 2019, Windows Server 2012, Windows 8.1, Windows Server 2016, Windows Server 2008 R2, Windows 10, Windows 10 Servers.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 8,000 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 14 tokensmean: 20.67 tokensmax: 34 tokens	min: 22 tokensmean: 82.34 tokensmax: 256 tokens

• Samples:

  anchor	positive
  Represent this sentence for searching relevant passages: Why are image titles causing issues in My Album Gallery?	The My Album Gallery plugin for WordPress is vulnerable to Stored Cross-Site Scripting via image titles in all versions up to, and including, 1.0.4. This is due to insufficient input sanitization and output escaping on the 'attachment->title' attribute. This makes it possible for authenticated attackers, with Author-level access and above, to inject arbitrary web scripts in pages that will execute whenever a user accesses an injected page.
  Represent this sentence for searching relevant passages: jenkins bitbucket server integration csrf bypass	Jenkins Bitbucket Server Integration Plugin 2.1.0 through 4.1.3 (both inclusive) allows attackers to craft URLs that would bypass the CSRF protection of any target URL in Jenkins.
  Represent this sentence for searching relevant passages: Android keystore integer overflow issue	Multiple integer overflows in the Blob class in keystore/keystore.cpp in Keystore in Android before 5.1.1 LMY48M allow attackers to execute arbitrary code and read arbitrary Keystore keys via an application that uses a crafted blob in an insert operation, aka internal bug 22802399.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1.0
• learning_rate: 2e-05
• warmup_steps: 0.1
• gradient_accumulation_steps: 2
• fp16: True
• per_device_eval_batch_size: 16
• load_best_model_at_end: True
• dataloader_drop_last: True

All Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1.0
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 2
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 4.5 days
• Evaluation: 30.9 minutes
• Total: 4.5 days

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 5.5.1
• Transformers: 5.10.2
• PyTorch: 2.12.0+cu126
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}