SentenceTransformer

This is a sentence-transformers model trained on the mnri_dataset and contrastive_dataset datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer

• Maximum Sequence Length: 2500 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • mnri_dataset
  • contrastive_dataset

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Qwen3Model'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 12\nSex: Male\nCancer type: Mediastinal mixed germ cell tumor\nHistology: Embryonal carcinoma, choriocarcinoma, and yolk sac tumor\nCurrent extent: Metastatic (mediastinal mass, mediastinal lymphadenopathy, and brain metastases in the right frontal and left parietal lobes). Most recent tumor markers (2018-03-30): AFP 310 ng/mL, Beta-hCG 850 mIU/mL, LDH 410 U/L; increasing trend.\nBiomarkers: SALL4 positive, AFP positive (yolk sac), Beta-hCG positive (choriocarcinoma), CD30 positive (embryonal carcinoma), Glypican-3 positive (yolk sac), Cytokeratin (AE1/AE3) positive. NGS: KRAS p.G12D pathogenic mutation, PTEN homozygous deletion, TP53 p.P72R (variant of uncertain significance), isochromosome 12p, gain of 20q.\nTreatment history:\n# 2015: bleomycin, etoposide, cisplatin; achieved partial response\n# 2016-02-09: Partial resection of anterior mediastinal mass\n# 2016-12-05-present: ifosfamide, cisplatin; achieved partial response, subsequently progressed',
]
documents = [
    'Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 1 to 21 years. Sex allowed: Both. Cancer type allowed: mediastinal mixed germ cell tumor. Histology allowed: pediatric malignant solid tumor. Cancer burden allowed: high-risk (expected 5-year Event-Free Survival < 60%) and < 30% chance at long-term cure and disease status of Stable Disease or better. Prior treatment required: prior primary therapy. Prior treatment excluded: NA. Biomarkers required: satisfactory TIL cellular product availability (assessed during screening). Biomarkers excluded: NA.',
    'Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 50 years. Sex allowed: Male. Cancer type allowed: Any cancer. Histology allowed: Hematologic. Cancer burden allowed: Any. Prior treatment required: NA. Prior treatment excluded: NA. Biomarkers required: NA. Biomarkers excluded: NA.',
    'Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 29 years. Sex allowed: both. Cancer type allowed: mixed phenotype acute leukemia with T cell dominant phenotype. Histology allowed: mixed phenotype acute leukemia with T cell dominant phenotype. Cancer burden allowed: Second or greater relapse or first relapse post-allogeneic hematopoietic stem cell transplant or primary refractory disease or relapsed and refractory disease. Prior treatment required: NA. Prior treatment excluded: CD7 targeted therapy. Biomarkers required: CD7 expression on leukemic blasts (at least 90% of blasts positive by flow cytometry or immunohistochemistry) assessed during screening. Biomarkers excluded: NA.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4132, 0.2552, 0.1695]])

Training Details

Training Datasets

mnri_dataset

• Dataset: mnri_dataset
• Size: 1,174,581 training samples
• Columns: patient_summary_trunc and this_space_trunc
• Approximate statistics based on the first 1000 samples:

  	patient_summary_trunc	this_space_trunc
  type	string	string
  details	min: 146 tokensmean: 323.8 tokensmax: 554 tokens	min: 101 tokensmean: 155.35 tokensmax: 283 tokens

• Samples:

  patient_summary_trunc	this_space_trunc
  Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 6Sex: MaleCancer type: Kaposiform hemangioendothelioma of the right lower extremityHistology: Spindle cell vascular neoplasm (CD31 positive, CD34 positive, ERG positive, HHV-8 negative)Current extent: Localized; status post wide local excision on 2023-02-06 with negative marginsBiomarkers: PIK3CA p.E542K mutant; PTEN, AKT1, KRAS, and TP53 wild-typeTreatment history:# 2020-10-28 to 2021-04-04: vincristine; achieved partial response (tumor decreased from 7 cm to 5.2 cm)# 2021-11-29 to 2022-10-18: paclitaxel; achieved partial response (tumor decreased from 9 cm to 6.4 cm)# 2023-02-06: wide local excision of right calf mass; negative margins	Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 14 years. Sex allowed: Male or Female. Cancer type allowed: Kaposiform hemangioendothelioma. Histology allowed: NA. Cancer burden allowed: NA. Prior treatment required: NA. Prior treatment excluded: sirolimus or other mTOR inhibitors. Biomarkers required: NA. Biomarkers excluded: NA.
  Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 14Sex: MaleCancer type: Kaposiform Hemangioendothelioma of the right lateral neckHistology: Spindle cell proliferation with kaposiform architectureCurrent extent: Stage T1N0M0; stable 3.1 cm mass in the right lateral neck as of 2021-02-08; platelets 228,000/uL as of 2020-12-12 (stable, no Kasabach-Merritt phenomenon)Biomarkers: CD31 positive, CD34 positive, ERG positive, HHV-8 negative, GLUT1 negative/patchy, Ki-67 less than 5%, PIK3CA p.H1047R mutation, PTEN p.S167R mutation, STK11 wild-type, TSC1 wild-typeTreatment history:# 2018-11-21-2021-02-08: observation; stable disease	Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 14 years. Sex allowed: Male or Female. Cancer type allowed: Kaposiform hemangioendothelioma. Histology allowed: NA. Cancer burden allowed: NA. Prior treatment required: NA. Prior treatment excluded: sirolimus or other mTOR inhibitors. Biomarkers required: NA. Biomarkers excluded: NA.
  Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 17
  Sex: Male
  Cancer type: T-cell lymphoblastic lymphoma
  Histology: T-cell lymphoblastic lymphoma
  Current extent: Relapsed and refractory. As of 2019-12-06, anterior mediastinal mass measuring 5.5 cm with associated superior vena cava narrowing and bilateral hilar lymphadenopathy. As of 2020-04-21, bone marrow involvement with 22% lymphoblasts. LDH is elevated (normalized during remission, rose to 412 U/L at relapse in 2019-03-26, and remains elevated as of 2020-08-21).
  Biomarkers: TdT positive, CD3 positive, CD7 positive, CD2 positive, CD5 positive, CD19 negative, CD20 negative, CD45 positive, Ki-67 >95%; NOTCH1 p.V1744G, CDKN2A homozygous deletion, PTEN p.R130Q, 12q gain
  Treatment history:

  2016-07 to 2016-12: vincristine, prednisone, L-asparaginase; best response ... | Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 29 years. Sex allowed: both. Cancer type allowed: T-cell acute lymphoblastic leukemia. Histology allowed: T-cell acute lymphoblastic leukemia. Cancer burden allowed: Second or greater relapse or first relapse post-allogeneic hematopoietic stem cell transplant or primary refractory disease or relapsed and refractory disease. Prior treatment required: NA. Prior treatment excluded: CD7 targeted therapy. Biomarkers required: CD7 expression on leukemic blasts (at least 90% of blasts positive by flow cytometry or immunohistochemistry) assessed during screening. Biomarkers excluded: NA. |

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

contrastive_dataset

• Dataset: contrastive_dataset
• Size: 1,824,636 training samples
• Columns: patient_summary_trunc, this_space_trunc, and label
• Approximate statistics based on the first 1000 samples:

  	patient_summary_trunc	this_space_trunc	label
  type	string	string	float
  details	min: 76 tokensmean: 294.03 tokensmax: 525 tokens	min: 101 tokensmean: 132.43 tokensmax: 179 tokens	min: -1.0mean: -0.7max: 1.0

• Samples:

  patient_summary_trunc	this_space_trunc	label
  Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 6Sex: MaleCancer type: Kaposiform hemangioendothelioma of the right lower extremityHistology: Spindle cell vascular neoplasm (CD31 positive, CD34 positive, ERG positive, HHV-8 negative)Current extent: Localized; status post wide local excision on 2023-02-06 with negative marginsBiomarkers: PIK3CA p.E542K mutant; PTEN, AKT1, KRAS, and TP53 wild-typeTreatment history:# 2020-10-28 to 2021-04-04: vincristine; achieved partial response (tumor decreased from 7 cm to 5.2 cm)# 2021-11-29 to 2022-10-18: paclitaxel; achieved partial response (tumor decreased from 9 cm to 6.4 cm)# 2023-02-06: wide local excision of right calf mass; negative margins	Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 14 years. Sex allowed: Male or Female. Cancer type allowed: Kaposiform hemangioendothelioma. Histology allowed: NA. Cancer burden allowed: NA. Prior treatment required: NA. Prior treatment excluded: sirolimus or other mTOR inhibitors. Biomarkers required: NA. Biomarkers excluded: NA.	-0.19999999999999996
  Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 14Sex: MaleCancer type: Kaposiform Hemangioendothelioma of the right lateral neckHistology: Spindle cell proliferation with kaposiform architectureCurrent extent: Stage T1N0M0; stable 3.1 cm mass in the right lateral neck as of 2021-02-08; platelets 228,000/uL as of 2020-12-12 (stable, no Kasabach-Merritt phenomenon)Biomarkers: CD31 positive, CD34 positive, ERG positive, HHV-8 negative, GLUT1 negative/patchy, Ki-67 less than 5%, PIK3CA p.H1047R mutation, PTEN p.S167R mutation, STK11 wild-type, TSC1 wild-typeTreatment history:# 2018-11-21-2021-02-08: observation; stable disease	Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 14 years. Sex allowed: Male or Female. Cancer type allowed: Kaposiform hemangioendothelioma. Histology allowed: NA. Cancer burden allowed: NA. Prior treatment required: NA. Prior treatment excluded: sirolimus or other mTOR inhibitors. Biomarkers required: NA. Biomarkers excluded: NA.	-0.19999999999999996
  Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age: 46Sex: FemaleCancer type: Retroperitoneal cancerHistology: Epithelioid HemangioendotheliomaCurrent extent: Metastatic (retroperitoneal mass, liver metastases, and ascites); progressive disease as of 2018-04-09Biomarkers: CD31 positive, ERG positive, CD34 patchy positive, Pan-Cytokeratin negative, S100 negative, HMB-45 negative, SMA negative, Ki-67 15%, WWTR1-YAP1 fusion, PIK3CA p.E545K mutation, RB1 hemizygous loss, STK11 p.T123M mutationTreatment history:# 2014-11-12: retroperitoneal debulking surgery; achieved no evidence of disease as of 2015-07-11# 2016-04-10-2017-05-16: paclitaxel; best response partial response (30 percent reduction in tumor diameter)# 2017-05-16-2018-04-09: capecitabine; best response stable disease	Instruct: Given a cancer patient summary, retrieve clinical trial options that are reasonable for that patient; or, given a clinical trial option, retrieve cancer patients who are reasonable candidates for that trial. Age range allowed: 0 to 14 years. Sex allowed: Male or Female. Cancer type allowed: Kaposiform hemangioendothelioma. Histology allowed: NA. Cancer burden allowed: NA. Prior treatment required: NA. Prior treatment excluded: sirolimus or other mTOR inhibitors. Biomarkers required: NA. Biomarkers excluded: NA.	-1.0

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 10
• learning_rate: 2e-05
• warmup_steps: 0.01
• bf16: True

All Hyperparameters

• per_device_train_batch_size: 10
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.01
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 7.8 hours

Framework Versions

• Python: 3.13.13
• Sentence Transformers: 5.4.1
• Transformers: 5.8.0
• PyTorch: 2.11.0+cu130
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

CoSENTLoss

@article{10531646,
    author={Huang, Xiang and Peng, Hao and Zou, Dongcheng and Liu, Zhiwei and Li, Jianxin and Liu, Kay and Wu, Jia and Su, Jianlin and Yu, Philip S.},
    journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing},
    title={CoSENT: Consistent Sentence Embedding via Similarity Ranking},
    year={2024},
    doi={10.1109/TASLP.2024.3402087}
}