BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from SQAI/bge-embedding-model. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: SQAI/bge-embedding-model
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("SQAI/bge-embedding-model2")
# Run inference
sentences = [
    'errors.controllerFault.lowLoadCurrent',
    '"Can you provide me with the current status of the streetlight on \'street name\', specifically in relation to its voltage under load, whether it\'s lower than expected and how that might be indicating potential electrical issues? Could you also give me insight into the current drawn by the streetlight, whether or not the relay is currently on or off, and if there are any faults in the lux module that may affect light level sensing and control? Moreover, could you tell me the type of dimming schedule applied, the ambient light level detected in lux, the total energy consumed so far recorded in kilowatt-hours, and the lower voltage threshold for this streetlight\'s efficient operation?"',
    '"Can you show me the current status of the relay in the streetlights located at the X-coordinate grid, highlighting any faults in the lux module that might be affecting light level sensing and control? Also, could you provide information on the current dimming level of these streetlights in operation, the type of dimming schedule applied, and whether the voltage is within the upper limit considered safe and efficient for their operation?"',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,865 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 5 tokensmean: 7.68 tokensmax: 14 tokens	min: 17 tokensmean: 89.79 tokensmax: 187 tokens

• Samples:

  positive	anchor
  threshold.lowLoadVoltage	"What is the maximum current level above which it is considered unsafe for a specific streetlight in my area, what is the minimum longitude of the geographic area this streetlight covers, is this streetlight's control mode automated or manually controlled, also, can you provide the delta or width of the grid area occupied by this group of streetlights, what is the level of AC voltage supply to this streetlight, what's the lower voltage threshold below which this streetlight may not operate efficiently, how many times has this streetlight been switched on, what is the minimum operational voltage under load conditions, and finally, what is the latitude of this streetlight?"
  asset.id	"Could you please tell me the scheduled dimming settings for the string stored streetlights, troubleshoot why these streetlights remain on during daylight hours, and confirm if this could be due to sensor faults? Also, I'd like to know the identifier for the parent group to which this group of streetlights belongs, and the IMEI number of the streetlight device."
  errors.controllerFault.highPower	"Can you provide an analysis of the efficiency of power usage by examining the power factor of the streetlights, especially in areas of the grid with high Y-coordinates, highlight instances where power consumption is significantly higher than expected which may indicate faults, identify situations where voltage under load is above safe levels, and assess if there are any problems with our central control system's ability to manage streetlight groups?"

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 208 evaluation samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 5 tokensmean: 7.55 tokensmax: 14 tokens	min: 19 tokensmean: 90.69 tokensmax: 187 tokens

• Samples:

  positive	anchor
  log.controlModeSwitch	"Can you provide the control mode switch identifier used for changing the default dimming level set for a specific group of streetlights, identified by their unique identifier, considering the time taken for the streetlight to activate or light up from the command, and possibly troubleshoot why the power consumption is lower than expected which could be due to hardware issues, quite possibly due to the relay responsible for turning the streetlight on and off sticking?"
  errors.controllerFault.luxModuleFault	"Can you provide the timestamp of the last update to the threshold settings, and detail any faults in the lux module related to light level sensing and control for the streetlight on this specific street name? I also want to know the longitude of the streetlight. And also, can you tell me what type of dimming schedule is applied to the streetlight, the type of port used for its dimming controls, and the total energy it has consumed, recorded in kilowatt-hours. Lastly, could you also provide the timestamp of the recorded streetlighting error, and confirm the status of the relay responsible for turning this streetlight on and off, as I am suspecting it might be sticking?"
  threshold.lowLoadCurrent	"What is the maximum safe voltage under load conditions for the city's streetlights, and do we possess the necessary rights to link these streetlights for synchronized control? Could you provide me with the timestamp of the latest data or action performed by our streetlights, and tell me the lower lux level threshold at which we would need to consider additional lighting? How often does each streetlight send a data report in normal operation, and what is the minimum load current level where we might start seeing suboptimal functioning? Have we been experiencing any problems with managing groups of streetlights via the central control system? Also, has there been any instances where the current under load was excessively high, indicating possible overloads, or situations where the operation temperature was belo normal limits due to environmental conditions? Lastly, have there been any noted communication issues between the streetlight's driver and the control system?"

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          384,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-06
• weight_decay: 0.03
• num_train_epochs: 200
• lr_scheduler_type: cosine
• warmup_ratio: 0.2
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• learning_rate: 2e-06
• weight_decay: 0.03
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 200
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.2
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.1.2+cu121
• Accelerate: 0.31.0
• Datasets: 2.19.1
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}