SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("himanshu23099/bge_embedding_finetune1")
# Run inference
sentences = [
    'Tourists visit reason',
    'What is All Saints Cathedral, and why is it architecturally significant?\nAll Saints Cathedral, locally known as Patthar Girja (Stone Church), is a renowned Anglican Christian Church located on M.G. Marg, Allahabad. Built in the late 19th century, it is one of the most beautiful and architecturally significant churches in Uttar Pradesh, attracting both tourists and pilgrims.',
    "What attractions are closest to the city center?\nNear the city center, you’ll find several attractions within a short distance. Anand Bhavan and Swaraj Bhavan are centrally located and offer insights into the Nehru family and India’s freedom movement. All Saints’ Cathedral, a magnificent Gothic-style church also known as the “Patthar Girja,” is located in Civil Lines and is one of Prayagraj's architectural gems. Company Bagh, a peaceful park, is also close by and ideal for a quiet stroll. Chandrashekhar Azad Park and Khusro Bagh are both centrally located as well, providing green spaces along with historical importance.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: val_evaluator
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 3,507 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 4 tokensmean: 11.76 tokensmax: 32 tokens	min: 8 tokensmean: 116.82 tokensmax: 504 tokens	min: 19 tokensmean: 121.15 tokensmax: 424 tokens

• Samples:

  anchor	positive	negative
  Where are the shuttle bus pickup points located within the Kumbh Mela grounds?	No, shuttle buses will not have dedicated volunteers specifically, but for assistance, you can reach out to the nearest information center.	The ancient art of weaving has captivated many cultures worldwide. In some regions, artisans use intricate patterns to tell stories, while others focus on vibrant colors that highlight their heritage. Experimentation with different materials can yield unique textures, adding depth to the final product. Workshops often provide insights into traditional techniques, ensuring these skills are passed down through generations.
  Hotel Ilawart start place	Is hotel pickup and drop-off available for the tours? Fixed pickup points, such as Hotel Ilawart, are provided for all tours. In some cases, pickup and drop-off can be arranged for locations within a 5 km radius of the starting point, but you must confirm this with the tour operator at the time of booking.	What all is included in the trip package?The trip package typically includes transportation, tour guide services, and breakfast. Meals such as lunch and dinner can be purchased separately. Hotel bookings are usually not included in the package, so you will need to arrange accommodation independently.
  Are there food stalls or restaurants at the Railway Junction that cater to dietary restrictions for pilgrims?	Yes, there are food stalls and restaurants available at the Railway Junction that cater to various dietary needs, including vegetarian and other dietary restrictions suitable for pilgrims.	The sound of the ocean waves rhythmically crashing against the shore creates a soothing symphony that invites relaxation. Seagulls soar above, occasionally diving down to catch a glimpse of fish beneath the surface. Beachgoers spread out their colorful towels, soaking up the sun's golden rays while children build sandcastles, their laughter mingling with the salty breeze. A distant sailboat glides across the horizon, hinting at adventures beyond the vast expanse of blue. As the sun sets, the sky transforms into a canvas of vibrant hues, signaling the end of another beautiful day by the sea.

• Loss: GISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01}
  

Evaluation Dataset

Unnamed Dataset

• Size: 877 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 877 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 5 tokensmean: 12.21 tokensmax: 32 tokens	min: 3 tokensmean: 115.93 tokensmax: 471 tokens	min: 15 tokensmean: 118.09 tokensmax: 422 tokens

• Samples:

  anchor	positive	negative
  Ganga bath benefit	What is the ritual of Snan or bathing? Taking bath at the confluence of Ganga, Yamuna and invisible Saraswati during Mahakumbh has special significance. It is believed that by bathing in this holy confluence, all the sins of a person are washed away and he attains salvation. Bathing not only symbolizes personal purification, but it also conveys the message of social harmony and unity, where people from different cultures and communities come together to participate in this sacred ritual. It is considered that in special circumstances, the water of rivers also acquires a special life-giving quality, i.e. nectar, which not only leads to spiritual development along with purification of the mind, but also gives physical benefits by getting health. List of Aliases: [['Snan', 'bathing'], ]	What benefits will I get by attending the Kumbh Mela?It is believed that bathing in the holy rivers during this time washes away sins and grants liberation from the cycle of life and death. Attending the Kumbh and taking a dip in the sacred rivers provides a unique opportunity for spiritual growth, purification, and selfrealization. ✨
  Guide provide what	What is the guide-to-participant ratio for each tour? Each tour is led by one guide per group, ensuring a personalized experience with ample opportunity for detailed insights and engagement. The guide will provide context, historical background, and answer any questions during the tour, offering a rich, informative experience for participants.	How many people can join a group tour?Group sizes depend on the type of vehicle selected. For instance, a Dzire accommodates up to 4 people, an Innova is suitable for 5-6 people, and larger groups (minimum 10 people) can travel in a Tempo Traveller. For even larger groups, multiple vehicles can be arranged to ensure everyone can travel together comfortably.
  How many rules must a Kalpvasi observe?	A Kalpvasi must observe 21 rules during Kalpvas, involving disciplines of the mind, speech, and actions.	The dancing colors of autumn leaves create a tapestry of nature’s beauty, inviting every eye to witness the grandeur of the changing seasons. Every gust of wind carries a whisper of nostalgia as trees shed their vibrant garments.

• Loss: GISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01}
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• gradient_accumulation_steps: 2
• learning_rate: 1e-05
• weight_decay: 0.01
• num_train_epochs: 30
• warmup_ratio: 0.1
• load_best_model_at_end: True

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 30
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.3.0
• Transformers: 4.46.2
• PyTorch: 2.5.1+cu121
• Accelerate: 1.1.1
• Datasets: 3.1.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

GISTEmbedLoss

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}