SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aaa961/finetuned-bge-m3-base-en")
# Run inference
sentences = [
    'Shell integration: bash and zsh don\'t serialize \\n and ; characters Part of https://github.com/microsoft/vscode/issues/155639\r\n\r\nRepro:\r\n\r\n1. Open a bash or zsh session\r\n2. Run:\r\n  ```sh\r\n  echo "a\r\n  … b"\r\n  ```\r\n  ![image](https://github.com/microsoft/vscode/assets/2193314/95ce755b-cec5-4da1-8791-75923a088efa)\r\n3. ctrl+alt+r to run recent command, select the last command, 🐛 it\'s run without the new line\r\n  ![image](https://github.com/microsoft/vscode/assets/2193314/98e81851-3a68-422c-94e8-e3e4bb21ef53)\r\n',
    'TreeView state out of sync Testing #117304\r\n\r\nRepro: Not Sure\r\n\r\nTest state shows passed in file but still running in tree view.\r\n\r\n![image](https://user-images.githubusercontent.com/6561887/108932683-a082c600-75fe-11eb-8ccd-afa5d0e2a91e.png)\r\n',
    'Setting icon and color in createTerminal API no longer works correctly See https://github.com/fabiospampinato/vscode-terminals/issues/77\r\n\r\nLooks like the default tab color/icon change probably regressed this.\r\n\r\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4264, 0.4315],
#         [0.4264, 1.0000, 0.4278],
#         [0.4315, 0.4278, 1.0000]])

Evaluation

Metrics

Triplet

• Dataset: bge-base-en-train
• Evaluated with TripletEvaluator

Triplet

• Dataset: bge-base-en-train
• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 336 training samples
• Columns: texts and label
• Approximate statistics based on the first 336 samples:

  	texts	label
  type	string	int
  details	min: 12 tokensmean: 340.9 tokensmax: 996 tokens	0: ~1.19%5: ~0.60%7: ~0.60%8: ~0.60%9: ~0.60%10: ~0.60%12: ~0.60%13: ~0.60%15: ~0.60%16: ~0.60%17: ~0.60%19: ~0.60%20: ~0.60%21: ~0.60%25: ~0.60%26: ~0.60%27: ~0.60%28: ~0.60%29: ~0.89%30: ~0.89%31: ~2.08%33: ~1.49%34: ~0.60%36: ~0.60%37: ~0.89%38: ~0.60%42: ~0.89%43: ~1.19%45: ~0.60%47: ~0.60%48: ~0.60%49: ~0.60%50: ~0.60%51: ~0.60%52: ~1.19%53: ~0.60%55: ~0.60%57: ~1.19%59: ~0.60%60: ~0.60%61: ~0.60%62: ~0.60%63: ~0.60%64: ~0.89%65: ~0.60%67: ~0.60%68: ~0.89%69: ~0.60%70: ~0.60%71: ~0.60%72: ~0.60%73: ~0.60%74: ~0.60%75: ~0.60%77: ~0.60%78: ~0.60%82: ~0.89%84: ~0.60%85: ~0.60%86: ~0.60%87: ~0.60%88: ~0.60%89: ~0.60%90: ~0.60%91: ~0.60%92: ~0.89%93: ~0.60%95: ~0.60%96: ~0.60%97: ~0.60%98: ~0.60%99: ~0.60%100: ~0.60%101: ~0.60%103: ~0.60%105: ~0.60%108: ~0.60%109: ~0.60%110: ~0.89%113: ~0.60%115: ~0.60%116: ~0.60%117: ~0.60%118: ~0.60%119: ~0.60%120: ~1.49%121: ~0.60%126: ~0.60%127: ~0.60%130: ~0.60%131: ~0.60%134: ~0.60%135: ~0.60%136: ~0.60%137: ~0.60%138: ~0.60%139: ~0.60%140: ~0.60%143: ~0.60%144: ~0.60%147: ~0.60%148: ~0.60%149: ~0.60%150: ~0.60%152: ~0.60%153: ~0.60%154: ~0.60%155: ~0.60%157: ~0.60%158: ~0.60%159: ~0.60%160: ~0.60%161: ~0.60%163: ~0.60%164: ~0.60%165: ~0.60%168: ~0.60%170: ~0.60%171: ~0.89%174: ~0.89%175: ~0.60%176: ~0.60%177: ~0.60%178: ~0.60%179: ~0.60%180: ~0.60%181: ~0.60%182: ~0.60%183: ~0.60%185: ~0.60%186: ~0.60%187: ~0.60%192: ~0.60%193: ~0.60%194: ~0.60%195: ~0.89%196: ~0.60%197: ~0.60%198: ~0.60%199: ~0.60%202: ~0.60%204: ~0.60%205: ~0.89%207: ~0.60%208: ~0.60%209: ~0.60%210: ~0.60%213: ~0.60%214: ~0.60%215: ~0.60%216: ~0.60%218: ~0.60%

• Samples:

  texts	label
  Branch list is sometimes out of order
  Type: Bug

• Loss: BatchSemiHardTripletLoss

Evaluation Dataset

Unnamed Dataset

• Size: 70 evaluation samples
• Columns: texts and label
• Approximate statistics based on the first 70 samples:

  	texts	label
  type	string	int
  details	min: 58 tokensmean: 303.57 tokensmax: 864 tokens	1: ~2.86%2: ~2.86%6: ~2.86%11: ~5.71%14: ~2.86%23: ~2.86%32: ~5.71%35: ~2.86%39: ~2.86%40: ~2.86%46: ~2.86%54: ~2.86%83: ~2.86%102: ~2.86%104: ~4.29%111: ~2.86%122: ~2.86%123: ~2.86%125: ~2.86%145: ~2.86%146: ~2.86%162: ~2.86%166: ~2.86%169: ~2.86%184: ~2.86%188: ~2.86%190: ~2.86%200: ~2.86%201: ~4.29%203: ~2.86%206: ~2.86%217: ~2.86%

• Samples:

  texts	label
  Ctrl+I stopped working after first hold+talk+release Testing #213355

• Loss: BatchSemiHardTripletLoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 1
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 1
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 5.1.0
• Transformers: 4.55.3
• PyTorch: 2.7.1+cu128
• Accelerate: 1.10.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

BatchSemiHardTripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}