SentenceTransformer based on FremyCompany/BioLORD-2023

U.S. openFDA Drug Product Research Model

This experimental model was fine-tuned on structured drug-product records from the official openFDA National Drug Code Directory. Training text includes brand name, active ingredients, strength, dosage form, and route.

The openFDA source data is generally available under CC0/public-domain terms; the base model and software dependencies retain their own terms. This model is for research and catalog-matching experiments. It must not be used for diagnosis, prescribing, dosage decisions, interaction checks, or automatic medicine substitution.

Held-out evaluation:

• nearest-neighbour exact-product accuracy: 0.6585 -> 0.8171
• validation triplet accuracy: 0.8995 -> 0.9416
• normalized separation margin: 2.5155 -> 2.8190

This is a sentence-transformers model finetuned from FremyCompany/BioLORD-2023 on the pairs and triplets datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: FremyCompany/BioLORD-2023
• Maximum Sequence Length: 32 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • pairs
  • triplets

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MPNetModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Brand: smart care island mango hand sanitizer. Ingredients: alcohol. Strength: 70 ml/100ml. Form: spray. Route: topical.',
    'Brand: ashleybelle moisturizing hand sanitizer cranberry sage. Ingredients: alcohol. Strength: 70 ml/100ml. Form: spray. Route: topical.',
    'Brand: walgreens dandruff itchy dry scalp defense anti-dandruff. Ingredients: pyrithione zinc. Strength: 10 mg/ml. Form: shampoo. Route: topical.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8526, 0.4529],
#         [0.8526, 1.0000, 0.4206],
#         [0.4529, 0.4206, 1.0000]])

Evaluation

Metrics

Composition

• Dataset: comp
• Evaluated with main.CompositionEvaluator

Triplet

• Dataset: val
• Evaluated with TripletEvaluator

Training Details

Training Datasets

pairs

• Dataset: pairs
• Size: 13,331 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 27 tokensmean: 31.86 tokensmax: 32 tokens	min: 28 tokensmean: 31.86 tokensmax: 32 tokens

• Samples:

  anchor	positive
  Brand: gps topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: primo topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.
  Brand: bencocaine topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: tiger supply inc topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.
  Brand: safco sensicaine ultra topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: advance topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.

• Loss: CachedGISTEmbedLoss with these parameters:

  {
      "guide": "SentenceTransformer(None)",
      "temperature": 0.01,
      "mini_batch_size": 64,
      "mini_batch_num_tokens": null,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }
  

triplets

• Dataset: triplets
• Size: 27,325 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 27 tokensmean: 31.8 tokensmax: 32 tokens	min: 28 tokensmean: 31.88 tokensmax: 32 tokens	min: 28 tokensmean: 31.61 tokensmax: 32 tokens

• Samples:

  anchor	positive	negative
  Brand: candee caine topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: advance topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: 7 select oral pain maximum strength relief. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: topical.
  Brand: kolorz topical anesthetic blue rasberry. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: kolorz topical anesthetic triple mint. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: lubelife climax control delay. Ingredients: benzocaine. Strength: 7.5 g/100ml. Form: spray. Route: topical.
  Brand: quala topical anesthetic gel. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: kolorz topical anesthetic cotton candy. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental.	Brand: hurricaine topical anesthetic. Ingredients: benzocaine. Strength: 200 mg/g. Form: gel. Route: dental | periodontal.

• Loss: CachedGISTEmbedLoss with these parameters:

  {
      "guide": "SentenceTransformer(None)",
      "temperature": 0.01,
      "mini_batch_size": 64,
      "mini_batch_num_tokens": null,
      "margin_strategy": "absolute",
      "margin": 0.0,
      "contrast_anchors": true,
      "contrast_positives": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 512
• num_train_epochs: 1.0
• learning_rate: 1e-05
• warmup_steps: 0.1
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

• per_device_train_batch_size: 512
• num_train_epochs: 1.0
• max_steps: -1
• learning_rate: 1e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 1.9 minutes
• Evaluation: 0.8 seconds
• Total: 1.9 minutes

Framework Versions

• Python: 3.12.10
• Sentence Transformers: 5.7.0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}