SentenceTransformer based on dunzhang/stella_en_1.5B_v5

This is a sentence-transformers model finetuned from dunzhang/stella_en_1.5B_v5. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: dunzhang/stella_en_1.5B_v5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 1536, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 1536, 'out_features': 1024, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Zybg/synthetic-clinical-embedding-model")
# Run inference
sentences = [
    ' Patient presents with LEFT hand pain following a fall while playing basketball 10 days ago. Impact directly on the LEFT hand during an attempt to break the fall. Immediate swelling and bruising noted over the 2nd and 3rd metacarpals. \n\nReported difficulty with grip strength and hand function, particularly with activities of daily living (ADLs) such as opening jars and typing on a keyboard. Unable to make a full fist and experiences pain (VAS 7/10) with any movement involving finger flexion and extension.\n\nBody chart:\n- Localized tenderness over the 2nd and 3rd metacarpal regions\n- Swelling present - moderate; no visible deformity noted\n- No neural symptoms; sensation intact across the hand\n- AROM: Restricted in MCP flexion (30°-40°), painful and limited in extension\n\nSHX:\nAge: 25\nOccupation: Graphic designer - predominantly desk work\nHobbies: Basketball, painting \n\nPatient was advised on RICE (Rest, Ice, Compression, Elevation) protocol initially. Incorporation of splint for support was initiated post-initial A&E visit. Currently scheduled for ORIF to stabilize the fracture. \n\nConcerns regarding recovery timeframe and return to sport expressed by the patient. Follow-up consultations planned post-surgery to assess for rehabilitation progress.\n\nNo previous history of fractures or orthopedic injuries. \n\n',
    'Condition name: Metacarpal Fracture ORIF\nBody region: Hand & Wrist\nRegion segments: {""}\nRegion sides: {Left,Right,Bilateral}\nDescription: A metacarpal fracture ORIF (Open Reduction and Internal Fixation) is a surgical procedure to repair a fracture in one of the metacarpal bones, which are the long bones in the hand located between the wrist and the fingers. This type of injury often occurs due to trauma, such as falls, sports injuries, or punching an object. ORIF is typically performed to realign the fractured bone fragments accurately and stabilize them with plates and screws, allowing for proper healing and restoration of function in the hand.\nSymptoms: Pain in the hand or wrist, Swelling and bruising, Deformity of the hand (e.g., angulated or rotated finger), Decreased range of motion, Difficulty in gripping or pinching, Numbness or tingling (if nerves are affected)\nMedical abbreviations: ORIF - Open Reduction and Internal Fixation, RICE - Rest, Ice, Compression, Elevation, ROM - Range of Motion, CT - Computed Tomography (if imaging is needed), X-ray - Radiographic imaging to assess the fracture\nIdentifying features: Common identifying features of a metacarpal fracture include localized swelling and tenderness over the affected metacarpal, a visible deformity such as a dorsal angulation of the finger, and limited finger mobility. X-rays are crucial in diagnosing the specific type and severity of the fracture.',
    'Condition name: Subtalar Fusion\nBody region: Foot & Ankle\nRegion segments: {""}\nRegion sides: {Left,Right,Bilateral}\nDescription: Subtalar fusion is a surgical procedure that involves the fusion of the subtalar joint, which is located below the ankle joint. This joint plays a critical role in foot motion and stability, allowing for inversion and eversion of the foot. The fusion is usually performed to alleviate chronic pain due to conditions such as subtalar joint arthritis, post-traumatic arthritis, or other degenerative conditions. By fusing the joint, the motion at that specific site is eliminated, which can help reduce pain and improve function in other parts of the foot and ankle.\nSymptoms: Chronic pain in the heel or ankle, Swelling around the subtalar joint, Stiffness in the foot, Limited range of motion in the ankle, Discomfort during activities such as walking or standing\nMedical abbreviations: STJ (Subtalar Joint), ORIF (Open Reduction and Internal Fixation), NSAIDs (Non-Steroidal Anti-Inflammatory Drugs), PT (Physical Therapy), ROM (Range of Motion)\nIdentifying features: Patients often exhibit significant limitations in foot mobility, especially in inversion and eversion movements. There may also be tenderness, swelling, and a history of previous foot or ankle injuries or surgeries.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,040 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 205 tokensmean: 283.37 tokensmax: 403 tokens	min: 195 tokensmean: 281.71 tokensmax: 470 tokens

• Samples:

  anchor	positive
  Patient presents following a fall at home 5 days ago, where they struck their face against a hard surface. Notable tenderness on the LEFT side of the face, particularly over the zygomatic arch and maxilla. Swelling evident in the LEFT periorbital region with some ecchymosis present.Reported severe pain when attempting to open the mouth, rated 7/10 at worst, and pain on palpation of the left maxillary sinus area. Limited range of motion (ROM) with mouth opening approximating to 30mm, rather than the normative 40-50mm. Exam reveals asymmetry with LEFT deviation of the mandible on opening. No neurological symptoms noted; however, patient does report episodes of tinnitus since the fall. Not feeling dizzy but has experienced some headaches over the past 3 days, exacerbated by bending forward. Does report some difficulty in eating solids due to pain, preferring soft foods. No history of dental trauma, and no changes in vision noted.Patient is currently using ice packs to manage sw...	Condition name: Facial FracturesBody region: Face/Head/JawRegion segments: {""}Region sides: {Left,Right,Bilateral,Central,N/A}Description: Facial fractures refer to breaks in the bones of the face and are commonly associated with trauma from accidents, falls, sports, or violence. These fractures can affect various parts of the face including the nasal bones, maxilla (upper jaw), mandible (lower jaw), zygomatic bones (cheekbones), and orbits (eye sockets). The extent of the fracture can range from a simple crack to complex fractures involving multiple facial bones.Symptoms: Facial swelling, Bruising around the eyes, nose, or cheeks, Pain at the fracture site, Deformity of the face, Difficulty opening or closing the mouth, Nasal bleeding or obstruction, Numbness or tingling in facial areas, Dental issues, such as loose or misaligned teethMedical abbreviations: FX - Fracture, MVA - Motor Vehicle Accident, CT - Computed Tomography, UT - Upper Teeth, LT - Lower Teeth, ZMC - Zygomatic...
  Presented with RIGHT elbow pain after a fall 2 weeks ago while playing basketball. Direct impact to the elbow upon landing. Immediate pain and swelling noted. Unable to fully extend or flex the elbow since the incident.Body chart:- Right lateral elbow tenderness- Limited ROM noted: Flexion is 30 degrees, Extension is -10 degrees from full extension.- Palpable bony deformity over the lateral aspect of the elbow.- NO neurovascular compromise; sensation intact in the radial, ulnar, and median nerve distributions.24hr symptom pattern: - Pain worsens with any movement or weight-bearing activities, particularly when trying to use the arm.- Night pain present, often wakes up due to discomfort.Aggs: Any attempts to lift or carry objects, especially with elbow flexion.Ease: Resting the arm in a sling and applying ice.SHX:- Age: 25- Occupation: Student-athlete- Regularly engaged in basketball (4-5 times/week).Reported no previous history of elbow issues. Has been taking OT...	Condition name: Elbow FractureBody region: ElbowRegion segments: {""}Region sides: {Left,Right,Bilateral}Description: An elbow fracture is a common injury that occurs when one of the bones in the elbow joint breaks. This can involve the humerus (the upper bone of the arm), the radius, or the ulna (the two bones of the forearm). Elbow fractures often result from falls, direct blows to the elbow, or accidents like sports injuries.Symptoms: Severe pain at the elbow, Swelling around the elbow joint, Inability to move the elbow or forearm, Deformity of the elbow (in cases of displaced fractures), Bruising around the elbow, Tenderness when touching the elbowMedical abbreviations: Fx (Fracture), Humeral Fracture (HF), Radius Fracture (RF), Ulna Fracture (UF), ORIF (Open Reduction Internal Fixation)Identifying features: Elbow fractures can be identified by localized pain, swelling, and bruising around the joint. Visible deformity may occur in more severe fractures. Diagnostic imaging, s...
  Patient presents with a complaint of pelvic heaviness and pressure, ongoing for approximately 2 months. Notable increase in symptoms during prolonged standing and following physical activity, such as walking and lifting. States feeling “a bulging sensation” in the vaginal area, particularly after being active. Hesitant to engage in high-impact exercises due to fear of exacerbating symptoms. Reports some difficulty initiating urination but no evidence of dysuria or hematuria. Experiences occasional urinary incontinence, especially with coughing or sneezing. Reports increased frequency of urination, needing to void every 1-2 hours during the day, but no nocturia. Menstrual history is regular, with LMP 1 week ago. Noted heightened symptomatology during menstruation. Denies any significant lower abdominal pain or discomfort. Body chart reflects pelvic discomfort, with no radiation of pain to the back or legs. No bowel issues reported. Sleep quality is fair; occasionally disturbed ...	Condition name: Pelvic Organ Prolapse (POP)Body region: PelvisRegion segments: {S1,S2,S3,S4,S5,Coccyx}Region sides: {Left,Right,Central,Bilateral,N/A}Description: Pelvic Organ Prolapse (POP) is a condition where pelvic organs such as the bladder, uterus, or rectum descend from their normal positions into the vagina due to weakness in the pelvic floor muscles and connective tissue. This can lead to physical discomfort and may cause urinary, bowel, and sexual function issues.Symptoms: A feeling of heaviness or pressure in the pelvic region, A bulge or protrusion in the vagina, Difficulty in urination or incomplete emptying of the bladder, Urinary incontinence (leakage), Bowel problems such as constipation or straining, Pain during sexual intercourse, Lower back painMedical abbreviations: POP (Pelvic Organ Prolapse), UI (Urinary Incontinence), PELVIC (Pelvic Exam and Lift for the Injured Chamber)Identifying features: The identification of POP typically includes physical examination...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 260 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 260 samples:

  	anchor	positive
  type	string	string
  details	min: 205 tokensmean: 288.07 tokensmax: 389 tokens	min: 202 tokensmean: 281.85 tokensmax: 425 tokens

• Samples:

  anchor	positive
  3 weeks post-operative from LEFT Total Hip Arthroplasty. Patient reports persistent pain in the LEFT hip region, particularly around the greater trochanter and along the lateral thigh. Pain is rated 5/10 at rest and increases to 7/10 with ambulation or transitional movements (e.g., standing up from sitting, going from lying to sitting). Subjective findings indicate mild swelling at the surgical site, although incision appears clean with no signs of infection. Using a walker for ambulation but is more confident in partial weight-bearing. Patient reports difficulty with activities of daily living (ADLs), particularly dressing and putting on shoes. Struggles to find a comfortable sleeping position, often waking during the night due to discomfort. Aggs: Weight-bearing activities and twisting movements. Ease: Rest and avoiding affected movements. Is currently on a regimen of pain medication (celecoxib) and reports fair control with occasional breakthrough pain. Previous history ...	Condition name: Total Hip Replacement (Total Hip Arthroplasty)Body region: HipRegion segments: {""}Region sides: {Left,Right,Bilateral}Description: Total Hip Replacement (THR), also known as Total Hip Arthroplasty, is a surgical procedure in which a damaged hip joint is replaced with a prosthetic implant. This condition often arises due to severe arthritis, fractures, or other degenerative diseases that result in pain and loss of function in the hip joint.Symptoms: Severe hip pain that limits daily activities, Stiffness in the hip joint, Swelling and inflammation in the hip area, Decreased range of motion in the hip, Difficulty walking or bearing weight on the affected sideMedical abbreviations: THR, THA, OA (Osteoarthritis), RA (Rheumatoid Arthritis), AVN (Avascular Necrosis)Identifying features: The procedure is characterized by surgical intervention to remove the damaged bone and cartilage of the hip joint and replace it with artificial components, typically made of metal, pl...
  Patient underwent LEFT hip resurfacing 3 weeks ago. Post-op period has included moderate pain localized to the LEFT hip, particularly exacerbated with weight-bearing activities and transitions from sitting to standing. Initial recovery was uneventful, but in the past week, increased discomfort has been noted, especially when walking longer distances or attempting stairs. Describes pain level as 6/10 during activity and relieved to 2/10 with rest.Body chart: - LEFT hip pain radiating to lateral thigh - NO referred pain down leg or to the groin - NO swelling around the hip joint 24-hour pattern: - More intense pain towards the evening after activity - Sleep disrupted due to discomfort when rolling onto LEFT side Aggs: - Weight-bearing activities—prolonged standing or walking > 10 minutes - Rising from sitting positions (e.g., getting up from chair) - Stairs are particularly painful Eases: - Resting and using ice - Elevation helps with swelling SHX: Active lifestyl...	Condition name: Hip ResurfacingBody region: HipRegion segments: {""}Region sides: {Left,Right,Bilateral}Description: Hip resurfacing is a type of hip surgery aimed at resurfacing the hip joint socket and femoral head primarily for the treatment of hip arthritis and to relieve pain. It involves placing a metal cap over the femoral head and a metal cup in the acetabulum, allowing for greater preservation of bone compared to traditional hip replacement. This procedure is often recommended for younger, more active patients who may place higher demands on their hip joint.Symptoms: Persistent hip pain during daily activities, Reduced range of motion in the hip joint, Stiffness in the hip area, Pain when walking, climbing stairs, or standing up from a seated position, Swelling in the hip jointMedical abbreviations: THR - Total Hip Replacement, HR - Hip Resurfacing, OA - Osteoarthritis, AR - Avascular Necrosis, ROM - Range of MotionIdentifying features: Typically indicated for younger p...
  6 weeks post hip arthroscopy for labral repair --> still experiencing persistent pain at the ANTERIOR hip/groin area. Pain began as a dull ache but has progressed to a sharp sensation, particularly during ambulation and when seated for prolonged periods.Reports intermittent swelling around the hip joint, especially after physical activity. Noted a significant increase in discomfort with activities such as rising from a seated position, stair climbing, and putting on shoes. Body chart indicates pain radiating only to the medial thigh, with occasional clicking sensations during hip flexion and extension. Has not noticed any knee symptoms.24hr pain pattern: - AM: mild discomfort upon waking.- PM: pain intensifies throughout the day with increased activity, leading to difficulty sleeping at night.Aggs: prolonged sitting, weight-bearing activities, and hip flexion.Ease: rest and ice application provide temporary relief.SHX: Age: 35 Occupation: IT professional, desk job. Exe...	Condition name: Hip ArthroscopyBody region: HipRegion segments: {""}Region sides: {Left,Right,Bilateral}Description: Hip arthroscopy is a minimally invasive surgical procedure used to diagnose and treat various hip joint conditions by inserting a small camera (arthroscope) and surgical instruments through small incisions in the skin. This technique provides orthopedic surgeons with a better view of the internal structures of the hip joint without the need for large incisions. Common conditions treated with hip arthroscopy include labral tears, femoroacetabular impingement, loose bodies, and hip joint infection.Symptoms: Hip pain, often worsening with activity, Stiffness in the hip joint, Reduced range of motion, Swelling in the hip area, Locking or clicking sensations in the hip, Pain radiating to the groin or thighMedical abbreviations: FAI - Femoroacetabular Impingement, CAM - Cam-type Impingement, Pincer - Pincer-type Impingement, MR - Magnetic Resonance, OA - Osteoarthritis, ...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• num_train_epochs: 1
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.3.1
• Transformers: 4.47.0
• PyTorch: 2.1.0+cu118
• Accelerate: 1.2.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}