SentenceTransformer based on intfloat/e5-small-v2

This is a sentence-transformers model finetuned from intfloat/e5-small-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/e5-small-v2
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'query: If baby dey warm up but still dey shiver and cannot keep normal temperature, wetin next?',
]
documents = [
    'passage: FINDING ON HISTORY FINDINGS ON CLINICAL JUDGMENT ACTION EXAMINATION Cold hands and feet -Temperature less than 36.5oC Moderate Condition 1.Ensure baby is dry and in -Cold Stress 36.0 – 36.4oC clean dry clothing -Moderate hypothermia 32.0 2.Place baby in skin to skin – 35.9oC contact in a pre-warmed Severe hypothermia <32.0oC blouse/shirt opening at the front with the baby in a nappy, hat and socks 3.Wrap baby in warm blanket 4.Ensure that the room is warm (at least 25oC) and is without drafts 5.If temperature is improving but baby cannot maintain normal temperature, put baby in Kangaroo Mother Care (KMC). 6.If temperature does not improve after one hour of re- warming, REFER immediately to hospital 2.Yellow hands and feet - Jaundice, No fever Moderate Condition REFER immediately to hospital where phototherapy is possible. Health Education 1.Continue skin to skin contact between mother and baby. 2.Teach mother about Kangaroo Mother Care -KMC). 3.Exclusive breastfeeding. 4.Counsel family members to support mother go through process.',
    'passage: iv. Small baby •Weight is 2.5 kg or below Moderate condition 1.Encourage exclusive •Baby can suck breastfeeding. 2.Keep baby warm using Kangaroo Mother Care (KMC). 3.Review in 2 days. If improving, review weekly for 2 weeks. If no improvement, REFER. v. Birth marks •Small black or blue patches Moderate condition 1. Reassure mother or red or bluish red vi. Bleeding from circumcision • Bleeding with or without Moderate 1.Put pressure on side for 10 pallor minutes 2. Show mother how to maintain pressure 1-2 hours. 3. If bleeding does not stop, REFER appropriately. vii. Constipation Abdominal distention or no Mild/Moderate condition finding. 1.Encourage exclusive breastfeeding 2.Probe rectum to stimulate defecation If no improvement, REFER viii. Baby not feeding well, •No abnormality Mild condition 1.Examine mother’s breasts Crying always. •Baby is well Irritable and treat if necessary. Suspect maternal causes 2.Observe how mother feeds infant and correct if necessary. Counsel mother accordingly.',
    "passage: MODERATE ACUTE MALNUTRITION: WFH/L greater than -3 to less than -2 z-scores OR MUAC 11.5 to 12.4cm. Treatment: If growth is faltering for 2 consecutive months, give Vitamin A, assess further or refer to hospital. Assess the child's feeding and counsel the mother on feeding recommendations. Give Albendazole if child is 1 year or older and has not had a dose in the previous 6 months. If feeding problems, follow up in 14 days. AT RISK OF ACUTE MALNUTRITION: Static weight or losing weight, -2 to less than -1 Z-Score. For age 6 months up to 59 months MUAC 12.5 to 13.5cm. Treatment: If child is less than 2 years old, and has growth faltering, assess feeding and counsel mother on feeding recommendations. Give Albendazole if child is 1 year or older and has not had a dose in the previous 6 months. Follow up in 14 days. If feeding problem, follow-up in 5 days. NO ACUTE MALNUTRITION: WFH/L greater than -1 z-scores OR MUAC greater than 13.5 cm.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 384] [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5057, 0.2280, 0.0387]])

Evaluation

Metrics

Information Retrieval

• Dataset: medichat-golden-eval
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "query_prompt": "query: ",
      "corpus_prompt": "passage: "
  }
  

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,709 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 8 tokensmean: 24.89 tokensmax: 58 tokens	min: 93 tokensmean: 258.62 tokensmax: 512 tokens	min: 69 tokensmean: 286.18 tokensmax: 512 tokens

• Samples:

  anchor	positive	negative
  query: Madam, patient get chest pain with nausea and black stool. Pulse na 45. How I take am?	passage: I. Chest pain with any of the Pulse < 50/min Or > 100/min Severe condition REFER immediately to the following: BP > 180/110mmHg appropriate health facility. Nausea Difficult breathing Vomiting Pain radiates to neck, arm, shoulder and back II. Chest pain BP <90/60mmHg Temperature Severe condition 1. IV Normal saline 500ml, stop if 37.2°C breathing becomes worse 2. IV/IM Ceftriaxone 1g stat If no improvement, REFER appro- priately.. III. Chest pain becomes Severe pain Severe condition REFER immediately to the appro- worse on lying down or deep priate health facility. breathing IV. Severe chest pain and In pains Severe condition Low- Osmolarity ORS or known sickle cell disease IV Normal Saline 500ml 6hourly REFER if no improvement appro- priately. V. Central chest pain, radiates Central chest pain Severe condition (suspect REFER immediately to the to the neck, jaw and arm that angina) appropriate health facility. is : recurrent attack (joint pain) resulting from physical activity...	passage: 1.How long have you had this complaint? Is this the first time? 1. General appearance: Ill-looking 2.Have you had a good appetite? Any nausea or vomiting? 2. Temperature, pulse respiration, blood pressure: check and 3.Have you been urinating more frequently at night? Any pain on record micturition? 3. Face: Puffy 4.Any bleeding from nose? 4. Eyes:pallor jaundice * Vomiting or coughing blood? 5. Chest: crepitation, wheezes, decreased breath sounds * Blood in the urine or stool? Black stool? 6. Heart: rate, rhythm, 5.Any cough? Or breathlessness at night? 7. Abdomen: large tender liver, enlarged spleen 6.Do you sleep with extra pillow? 8. Limbs: a) legs 7.Have you ever been treated for jaundice? i) for oedema ii) does swelling pit on pressure (if not pitting, think of ele- phantiasis) b) hands for oedema 10. Urine: check for protein and sugar FINDING ON HISTORY FINDINGS ON CLINICAL ACTION EXAMINATION JUDGEMENT I. Swollen legs with large Swollen legs Severe condition REFER immedi...
  query: What is the correct Ribavir dose for a child weighing 3kg	passage: Hepatitis: Diffuse inflammation of the liver that can be caused by infective agents, drugs, alcohol, autoimmune and toxins. Clinical Features: Acute hepatitis (lasting for less than 6 months): Mild-to-moderate jaundice, Vague right upper quadrant discomfort, Fever, Hepatomegaly. Chronic hepatitis (lasting more than 6 months): General malaise/Fatigue, Re-occurence of jaundice. Complications: Liver failure, Bleeding tendencies, Cirrhosis/Liver cancer. Investigations: Liver Function Tests, Serologic markers of Hepatitis A, B, C, D and E, Viral load, Abdominal ultrasonography, Prothrombin time, HIV Screening. Treatment Goals: Provide supportive measures, Prevent progression to chronic phase. Non-drug Treatment: High carbohydrate and Normal protein diet, Discontinuation of hepatotoxic medication, Bed rest. Drug Treatment: Chronic Hepatitis B — Treatment is required in patients with viral load exceeding 2000 IU/ml for HBeAg negative or 20000 IU/ml for HBeAg positive status. Pegylate...	passage: Medicine doses in children are generally based on body weight in kilograms or age ranges: first one month (neonate), up to 1 year (infant), 1-5 years, and 6-12 years. Unless age is specified, the term child includes persons aged 12 years and below. Doses are calculated based on body weight or body surface area in square metres. Body Surface Area estimates are more accurate for calculation of paediatric doses as many physiological phenomena correlate better to BSA. For most medicines the adult maximum dose should not be exceeded. For example if the dose is stated as 4 mg/kg with max 180 mg, a child weighing 10 kg should receive 40 mg but a child weighing 50 kg should receive 180 mg not 200 mg. Young children may require higher doses per kilogram than adults because of their higher metabolic rate. Calculation by body weight in an overweight child may result in much higher doses than necessary so doses should be calculated based on ideal body weight in relation to height and age....
  query: Aunty, girl dey bleed plenty during period with clots and she pale. Which care I go give for clinic?	passage: I. Abdominal pain with men- Lower abdominal tenderness, Moderate condition 1.Reassure the client/patient struation; Foul smelling High temperature Suspect infection 2. Counsel on personal hygiene discharge per vagina; Fever (use of sanitary pad) 3. Capsule Amoxicillin 250mg 8hourly for 5 days 4. Conduct RDT, if positive, treat for uncomplicated malaria 5. Review in 5 days, if no improvement REFER immediately to the appropriate health facility II. Heavy menstrual flow with May be pale Moderate condition 1.Reassure the client/patient clots, prolonged menstruation Blood clots per vagina Abnormal menstrual flow 2. Tablet Paracetamol 500mg 8hourly for 3 days III. Lower abdominal pain Lower abdominal Mild condition 1.Tablet Paracetamol 500 mg associated with menstruation tenderness, no fever, Suspect dysmenorrhea 8hourly for 3days No foul smelling Reassure and counsel discharge IV. Mucus or blood per vagina Normal mucus, no Mild condition 1. Counsel and reassure patient. smell Norma...	passage: 1.Have you used any contraceptive method before? General appearance: anxious thin, obese, ill, 2.What method were you using? Which Clinic did you attend? Skin: scars (signs of domestic violence) Why do you want to attend this particular clinic? Weight: weigh and record 3.Who referred you to this Clinic? Blood pressure: measure and record 4.How many children have you had? How many are alive? How Hair: hair loss especially in temporal region many dead? When did you have your last baby? Did you have a Face: swelling normal vaginal delivery or were you assisted (caesarean section, Eyes: pallor or jaundice forceps delivery or vacuum extraction). Neck: goitre, lymph nodes 5.Are you breastfeeding? Chest: shape, respiratory difficulty or rapid creps or wheeze 6.Are you menstruating now? When did you start? If not, when Heart: rate, rhythm, murmur was the last menstrual period? Breast: lump, dimpling of the skin, abnormal appearance of the 7.Do you normally have; nipples, retraction of...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• gradient_accumulation_steps: 2
• learning_rate: 2e-05
• num_train_epochs: 20
• warmup_steps: 0.1
• fp16: True
• load_best_model_at_end: True
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 20
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 29.2 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}