SentenceTransformer based on llm-semantic-router/mmbert-embed-32k-2d-matryoshka

This is a sentence-transformers model finetuned from llm-semantic-router/mmbert-embed-32k-2d-matryoshka. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: llm-semantic-router/mmbert-embed-32k-2d-matryoshka
• Maximum Sequence Length: 32768 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    "A 45-year-old woman, gravida 3, para 2, at 18 weeks' gestation comes to the physician for a prenatal visit. Ultrasonography at a previous visit when she was at 12 weeks' gestation showed a hypoplastic nasal bone. Pelvic examination shows a uterus consistent in size with an 18-week gestation. Maternal serum studies show low α-fetoprotein and free estriol concentrations, and increased inhibin A and β-hCG concentrations. Physical examination of the infant after delivery is most likely to show which of the following findings?",
    "A 45-year-old woman, gravida 3, para 2, at 18 weeks' gestation comes to the physician for a prenatal visit. Ultrasonography at a previous visit when she was at 12 weeks' gestation showed a hypoplastic nasal bone. Pelvic examination shows a uterus consistent in size with an 18-week gestation. Maternal serum studies show low α-fetoprotein and free estriol concentrations, and increased inhibin A and β-hCG concentrations. Physical examination of the infant after delivery is most likely to show which of the following findings?\n\nAnswer: Single transverse palmar crease",
    'A 35-year-old woman gravida 2, para 1, comes to the physician for her first prenatal visit. Pregnancy and delivery of her first child were uncomplicated. She is not sure about the date of her last menstrual period. Pelvic examination shows a uterus consistent in size with a 10-week gestation. An ultrasound examination confirms the gestational age and shows one fetus with no indication of multiple gestations. During counseling on pregnancy risks and possible screening and diagnostic tests, the patient states she would like to undergo screening for Down syndrome. She would prefer immediate and secure screening with a low risk to herself and the fetus. Which of the following is the most appropriate next step in management at this time?\n\nAnswer: Cell-free fetal DNA testing',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9805, 0.5273],
#         [0.9805, 1.0000, 0.5430],
#         [0.5273, 0.5430, 1.0000]], dtype=torch.bfloat16)

Training Details

Training Dataset

Unnamed Dataset

• Size: 492 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 492 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 36 tokensmean: 219.2 tokensmax: 791 tokens	min: 49 tokensmean: 227.77 tokensmax: 819 tokens	min: 51 tokensmean: 159.34 tokensmax: 430 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  A 48-year-old woman presents to the emergency department because of increasingly severe right upper abdominal pain, fever, and non-bloody vomiting for the last 5 hours. The pain is dull, intermittent, and radiates to her right shoulder. During the past 3 months, she has had recurring abdominal discomfort after meals. The patient underwent an appendectomy more than 30 years ago. She has hypertension, diabetes mellitus type 2, and chronic back pain. She takes bisoprolol, metformin, and ibuprofen daily. She is 171 cm (5 ft 6 in) tall and weighs 99 kg (218 lb). Her BMI is 35.2 kg/m2. She appears uncomfortable and is clutching her abdomen. Her temperature is 38.5°C (101.3°F), pulse is 108/min, and blood pressure is 150/82 mm Hg. Abdominal examination shows right upper quadrant abdominal tenderness and guarding. Upon deep palpation of the right upper quadrant, the patient pauses during inspiration. Laboratory studies show the following:BloodHemoglobin 13.1 g/dLLeukocyte count 10,900/mm3P...	A 48-year-old woman presents to the emergency department because of increasingly severe right upper abdominal pain, fever, and non-bloody vomiting for the last 5 hours. The pain is dull, intermittent, and radiates to her right shoulder. During the past 3 months, she has had recurring abdominal discomfort after meals. The patient underwent an appendectomy more than 30 years ago. She has hypertension, diabetes mellitus type 2, and chronic back pain. She takes bisoprolol, metformin, and ibuprofen daily. She is 171 cm (5 ft 6 in) tall and weighs 99 kg (218 lb). Her BMI is 35.2 kg/m2. She appears uncomfortable and is clutching her abdomen. Her temperature is 38.5°C (101.3°F), pulse is 108/min, and blood pressure is 150/82 mm Hg. Abdominal examination shows right upper quadrant abdominal tenderness and guarding. Upon deep palpation of the right upper quadrant, the patient pauses during inspiration. Laboratory studies show the following:BloodHemoglobin 13.1 g/dLLeukocyte count 10,900/mm3P...	A 27-year-old woman presents with acute abdominal pain in her right upper quadrant. The pain came on suddenly while she was eating dinner. After this pain she began feeling dizzy and came to the emergency department. In the ED, her blood pressure is 75/40 mmHg, pulse is 100/minute, and she is afebrile. On physical exam, she feels too light-headed to ambulate. She demonstrates normal bowel sounds with tenderness upon palpation in the right upper quadrant. The patient is deemed too unstable for imaging. An abdominal radiograph and CT are reviewed from a recent previous visit to the ED for mild abdominal pain, and are shown in Figures A and B, respectively. Which of the following specific additional findings in her history supports the most likely diagnosis?Answer: Use of oral contraceptives (OCPs) for birth control
  A 42-year-old man presents to his primary care provider for abdominal pain. He reports that for several months he has been experiencing a stabbing pain above the umbilicus during meals. He denies associated symptoms of nausea, vomiting, or diarrhea. The patient’s past medical history is significant for hypertension and hyperlipidemia for which he takes amlodipine and atorvastatin. His family history is significant for lung cancer in his father. The patient is a current smoker with a 20 pack-year smoking history and drinks 3-5 beers per week. Initial laboratory testing is as follows:Serum:Na+: 141 mEq/LK+: 4.6 mEq/LCl-: 102 mEq/LHCO3-: 25 mEq/LUrea nitrogen: 14 mg/dLCreatinine: 1.1 mg/dLGlucose: 120 mg/dLCalcium: 8.4 mg/dLAlkaline phosphatase: 66 U/LAspartate aminotransferase (AST): 40 U/LAlanine aminotransferase (ALT): 52 U/LGastrin: 96 pg/mL (<100 pg/mL)Lipase: 90 U/L (<160 U/L)The patient is started on a proton pump inhibitor without symptomatic improvement after 6 w...	A 42-year-old man presents to his primary care provider for abdominal pain. He reports that for several months he has been experiencing a stabbing pain above the umbilicus during meals. He denies associated symptoms of nausea, vomiting, or diarrhea. The patient’s past medical history is significant for hypertension and hyperlipidemia for which he takes amlodipine and atorvastatin. His family history is significant for lung cancer in his father. The patient is a current smoker with a 20 pack-year smoking history and drinks 3-5 beers per week. Initial laboratory testing is as follows:Serum:Na+: 141 mEq/LK+: 4.6 mEq/LCl-: 102 mEq/LHCO3-: 25 mEq/LUrea nitrogen: 14 mg/dLCreatinine: 1.1 mg/dLGlucose: 120 mg/dLCalcium: 8.4 mg/dLAlkaline phosphatase: 66 U/LAspartate aminotransferase (AST): 40 U/LAlanine aminotransferase (ALT): 52 U/LGastrin: 96 pg/mL (<100 pg/mL)Lipase: 90 U/L (<160 U/L)The patient is started on a proton pump inhibitor without symptomatic improvement after 6 w...	A 42-year-old man comes to his primary care physician complaining of abdominal pain. He describes intermittent, burning, epigastric pain over the past 4 months. He reports that the pain worsens following meals. He had an upper gastrointestinal endoscopy done 2 months ago that showed a gastric ulcer without evidence of malignancy. The patient was prescribed pantoprazole with minimal improvement in symptoms. He denies nausea, vomiting, diarrhea, or melena. The patient has no other medical problems. He had a total knee replacement 3 years ago following a motor vehicle accident for which he took naproxen for 2 months for pain management. He has smoked 1 pack per day since the age 22 and drinks 1-2 beers several nights a week with dinner. He works as a truck driver, and his diet consists of mostly of fast food. His family history is notable for hypertension in his paternal grandfather and coronary artery disease in his mother. On physical examination, the abdomen is soft, nondistended, and ...
  A 62-year-old woman comes to the physician in June for a routine check-up. She has chronic back pain and underwent an appendectomy at the age of 27. She is married and has two kids. The patient recently got back from a cruise to Mexico where she celebrated her 40th wedding anniversary. Her last mammogram was 6 months ago and showed no abnormalities. Her last Pap smear was 2 years ago and unremarkable. A colonoscopy 5 years ago was normal. Her mother died of breast cancer last year and her father has arterial hypertension. Her immunization records show that she has never received a pneumococcal or a shingles vaccine, her last tetanus booster was 6 years ago, and her last influenza vaccine was 2 years ago. She drinks 1– 2 alcoholic beverages every weekend. She takes a multivitamin daily and uses topical steroids. She regularly attends water aerobic classes and physical therapy for her back pain. She is 168 cm (5 ft 6 in) tall and weighs 72 kg (160 lb); BMI is 26 kg/m2. Her temperature is...	A 62-year-old woman comes to the physician in June for a routine check-up. She has chronic back pain and underwent an appendectomy at the age of 27. She is married and has two kids. The patient recently got back from a cruise to Mexico where she celebrated her 40th wedding anniversary. Her last mammogram was 6 months ago and showed no abnormalities. Her last Pap smear was 2 years ago and unremarkable. A colonoscopy 5 years ago was normal. Her mother died of breast cancer last year and her father has arterial hypertension. Her immunization records show that she has never received a pneumococcal or a shingles vaccine, her last tetanus booster was 6 years ago, and her last influenza vaccine was 2 years ago. She drinks 1– 2 alcoholic beverages every weekend. She takes a multivitamin daily and uses topical steroids. She regularly attends water aerobic classes and physical therapy for her back pain. She is 168 cm (5 ft 6 in) tall and weighs 72 kg (160 lb); BMI is 26 kg/m2. Her temperature is...	A 66-year-old G3P3 presents with an 8-year-history of back pain, perineal discomfort, difficulty urinating, recurrent malaise, and low-grade fevers. These symptoms have recurred regularly for the past 5–6 years. She also says that there are times when she experiences a feeling of having a foreign body in her vagina. With the onset of symptoms, she was evaluated by a physician who prescribed her medications after a thorough examination and recommended a vaginal pessary, but she was non-compliant. She had 3 vaginal deliveries She has been menopausal since 51 years of age. She does not have a history of malignancies or cardiovascular disease. She has type 2 diabetes mellitus that is controlled with diet and metformin. Her vital signs include: blood pressure 110/60 mm Hg, heart rate 91/min, respiratory rate 13/min, and temperature 37.4℃ (99.3℉). On physical examination, there is bilateral costovertebral angle tenderness. The urinary bladder is non-palpable. The gynecologic examination reve...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 2
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Framework Versions

• Python: 3.12.10
• Sentence Transformers: 5.2.2
• Transformers: 5.0.0
• PyTorch: 2.7.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}