SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5 on the bge-retrieval-data-finance-50_k dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • bge-retrieval-data-finance-50_k

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("40Hz/autoresearch-business-v1")
# Run inference
sentences = [
    'gmc is what type of truck',
    '- It has been said that the roots of GMC Trucks go back to the 3 Rs of early truck development: Rapid, Reliance and Randolph. Truly, the first GMC trucks were Rapid or Reliance trucks with GMC nameplates. However, Randolph trucks apparently had little or no influence on GMC design.',
    'business economic policy international africa house believes africans are worse Despite projects such as direct dividends, the gap between rich and poor is still worsened by natural resources. Investment from the profits of natural resources in human development is relatively low in Africa. In 2006, 29 of the 31 lowest scoring countries for HDI were in Africa, a symptom of low re-investment rates [1] . Generally it is only the economic elite who benefit from any resource extraction, and reinvestment rarely strays far from urban areas [2] . This increases regional and class inequality, ensuring poverty persists.\n\n[1] African Development Bank ‘African Development Report 2007’ pg.110 http://www.afdb.org/fileadmin/uploads/afdb/Documents/Publications/Maximizing%20the%20Benefits%20from%20-%20Oil%20and%20Gas%20in%20Africa.pdf\n\n[2] Ibid\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7964, 0.0219],
#         [0.7964, 1.0000, 0.0947],
#         [0.0219, 0.0947, 1.0000]])

Training Details

Training Dataset

bge-retrieval-data-finance-50_k

• Dataset: bge-retrieval-data-finance-50_k
• Size: 38,236 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 5 tokensmean: 18.69 tokensmax: 256 tokens	min: 30 tokensmean: 101.75 tokensmax: 256 tokens

• Samples:

  anchor	positive
  price chopper operating headquarters	Price Chopper cuts 80 jobs at corporate headquarters Exterior of the new Price Chopper Headquarters Building at 461 Nott St. on Thursday, June 3, 2010, in Schenectady, NY. The Green Building is LEED certified. (Luanne M. Ferris / Times Union) Exterior of the new Price Chopper Headquarters Building at 461 Nott St. on Thursday, June 3, 2010, in Schenectady, NY. The Green Building is LEED certified. Price Chopper has laid off 80 employees at its corporate headquarters, the chain announced Wednesday. The layoffs include management, field, clerical and administrative support workers, according to a statement from the company.
  vibrent health	- Vibrent Health is an innovative health technology company that uses evidence-based B2B2C technology infrastructure called Adaptive Platform for Personalized Engagement that enables longitudinal data collection, engagement and insights.
  what is the dry harvesting method of coffee cherries	Harvesting, Milling and Packing Coffee â Part 2 of âAll About Coffeeâ Coffee Milling: After harvesting, the coffee cherries must then be set to dry. The oldest and “natural” method of this process is the Dry Mill where the coffee cherries are cleaned and sorted before its exposure to the sun.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "mini_batch_num_tokens": null,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

bge-retrieval-data-finance-50_k

• Dataset: bge-retrieval-data-finance-50_k
• Size: 2,013 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 5 tokensmean: 21.01 tokensmax: 256 tokens	min: 27 tokensmean: 103.73 tokensmax: 256 tokens

• Samples:

  anchor	positive
  What is the name of the Australian-based visual effects company founded in 1995 in Adelaide whose spin-off privately held company was founded in April 2000 and has customers worldwide in film, advertising, etc.?	Rising Sun Research Cospective Pty Ltd is an Australian software firm. Originally born as Rising Sun Research, a technology spin-off from the boutique visual effects company Rising Sun Pictures, the company now operates under the Cospective banner to reflect the broadening nature of the business. Cospective has customers worldwide in film, advertising, television, games and design. The privately held company was founded in April 2000 and is headquartered in Adelaide, Australia.
  Who was born first, Takeshi Kitano or Arturo Ripstein?	Takeshi Kitano Takeshi Kitano (北野 武 , Kitano Takeshi , born 18 January 1947) is a Japanese comedian, television personality, director, actor, author, and screenwriter. While he is known primarily as a comedian and TV host in his native Japan, abroad he is known almost entirely for his filmwork. With the exception of his works as a film director, he is known almost exclusively by the stage name Beat Takeshi (ビートたけし , Bīto Takeshi ) .
  what year o reilly auto parts founded	O'Reilly Auto Parts O’Reilly Auto Parts is an American auto parts retailer that provides automotive aftermarket parts, tools, supplies, equipment, and accessories in the United States serving both the professional service providers and do-it-yourself customers. Founded in 1957 by the O’Reilly family, the company operates more than 4,829 stores in 47 states.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "mini_batch_num_tokens": null,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• learning_rate: 2e-05
• warmup_steps: 0.1
• fp16: True
• project: autoresearch-business
• trackio_space_id: 40Hz/autoresearch-gpu
• push_to_hub: True
• hub_model_id: 40Hz/autoresearch-business-v1
• batch_sampler: no_duplicates

All Hyperparameters

• per_device_train_batch_size: 32
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: autoresearch-business
• trackio_space_id: 40Hz/autoresearch-gpu
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: True
• hub_private_repo: None
• hub_model_id: 40Hz/autoresearch-business-v1
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Training Time

• Training: 20.5 minutes
• Evaluation: 23.5 seconds
• Total: 20.9 minutes

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.7.0
• Transformers: 5.15.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}