SentenceTransformer based on mixedbread-ai/mxbai-embed-large-v1

This is a sentence-transformers model finetuned from mixedbread-ai/mxbai-embed-large-v1. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: mixedbread-ai/mxbai-embed-large-v1
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("youssefkhalil320/mxbai-embed-large-v1-medical-nli_v1")
# Run inference
sentences = [
    "Given the patient's recent surgery and that the bleeding had stopped a colonoscopy was planned as an outpatient.",
    'Patient has significant PSH',
    'Patient has colon cancer',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: val-triplets
• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 7,603 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokensmean: 29.42 tokensmax: 256 tokens	min: 4 tokensmean: 9.38 tokensmax: 25 tokens	min: 4 tokensmean: 8.89 tokensmax: 21 tokens

• Samples:

  anchor	positive	negative
  O2: 94% 4Lnc.	The patient is on 4L of oxygen via nasal cannula	The patient’s oxygen saturation is 100% on room air
  The patient has received 500 mg of intravenously levofloxacin given at the outside hospital, and the patient has received intravenous vancomycin as well as ceftazidime in our Emergency Department.	The patient has received broad spectrum antibiotics.	The patient has pneumonia.
  Cardiac enzymes done at OSH showed CK 363, CK-MB 33, TropI 6.78.	The patient has cardiac ischemia.	The patient has normal cardiac perfusion.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.3
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 938 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 938 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokensmean: 31.03 tokensmax: 187 tokens	min: 4 tokensmean: 9.08 tokensmax: 21 tokens	min: 4 tokensmean: 8.76 tokensmax: 19 tokens

• Samples:

  anchor	positive	negative
  History of acute renal failure.	The patient has had kidney damage.	The patient has always had normal functioning kidneys.
  Of note, pt had recent workup for intermittent abd discomfort and bloating, CT abd showed cholelithiasis and endometrial thickening, due for endometrial biopsy with Gyn.	Patient has findings warranting biopsy on imaging	Patient has endometrial cancer
  She states she drinks about [2-17] glasses of wine per night, but also admits to drinking up to a full bottle of wine during the day when she is home alone.	the patient consumes alcohol	the patient denies alcohol use

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.3
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• weight_decay: 0.01
• num_train_epochs: 40.0
• warmup_ratio: 0.1
• load_best_model_at_end: True
• push_to_hub: True
• hub_model_id: youssefkhalil320/mxbai-embed-large-v1-medical-nli_v1

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 40.0
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: youssefkhalil320/mxbai-embed-large-v1-medical-nli_v1
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.19
• Sentence Transformers: 3.3.1
• Transformers: 4.44.2
• PyTorch: 2.8.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}