SentenceTransformer based on jhu-clsp/ettin-encoder-17m

This is a sentence-transformers model finetuned from jhu-clsp/ettin-encoder-17m on the msmarco-embed-dense_on dataset. It maps sentences & paragraphs to a 256-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: jhu-clsp/ettin-encoder-17m
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 256 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • msmarco-embed-dense_on

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Pooling({'embedding_dimension': 256, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    'Healthcare is involved, directly or indirectly, with the provision of health services to individuals. These services can occur in a variety of work settings, including hospitals, clinics, dental offices, out-patient surgery centers, birthing centers, emergency medical care, home healthcare, and nursing homes. Healthcare workers face a number of serious safety and health hazards.',
]
documents = [
    'In a right triangle, the orthocenter is the polygon vertex of the right angle. When the vertices of a triangle are combined with its orthocenter, any one of the points is the orthocenter of the other three, as first noted by Carnot (Wells 1991). These four points therefore form an orthocentric system. The circumcenter and orthocenter are isogonal conjugates. The orthocenter lies on the Euler line.',
    'who makes fire extinguishers',
    'what is powdery mildew cannabis',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 256] [3, 256]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5302, 0.5367, 0.4435]])

Evaluation

Metrics

Information Retrieval

• Dataset: NanoMSMARCO
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

msmarco-embed-dense_on

• Dataset: msmarco-embed-dense_on
• Size: 973,530 training samples
• Columns: text and label
• Approximate statistics based on the first 100 samples:

  	text	label
  type	string	list
  modality	text
  details	min: 4 tokensmean: 45.82 tokensmax: 179 tokens	size: 768 elements

• Samples:

  text	label
  how long background check bank returned	[-1.896484375, 0.288330078125, -0.406005859375, -1.6953125, 1.98046875, ...]
  Humira® (adalimumab) is a prescription medication licensed to treat certain inflammatory conditions that affect the joints, spine, or digestive system. Specific Humira uses include the treatment of: Ankylosing spondylitis. Crohn's disease. Juvenile idiopathic arthritis, also known as juvenile rheumatoid arthritis.	[-1.77734375, -0.072265625, -0.301513671875, -0.66748046875, 0.80810546875, ...]
  Bonfim is a municipality located in the mideast of the state of Roraima in Brazil. Its population is 12,626 and its area is 8,095 km². The city lies opposite the Takutu River from Lethem, Guyana. The Takutu River Bridge links Bonfim and Roraima with the town of Lethem and the Atlantic port of Georgetown, Guyana.	[-0.411865234375, 1.58984375, 0.76123046875, 1.6171875, 0.34375, ...]

• Loss: EmbedDistillLoss with these parameters:

  {
      "distance_metric": "cosine",
      "projection_dim": 768
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• learning_rate: 0.0001
• warmup_steps: 0.1
• weight_decay: 0.01
• bf16: True
• load_best_model_at_end: True
• seed: 12
• dataloader_num_workers: 2

All Hyperparameters

• per_device_train_batch_size: 32
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 0.0001
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 12
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 37.8 minutes
• Evaluation: 12.9 seconds
• Total: 38.0 minutes

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 5.6.1
• Transformers: 5.14.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

EmbedDistillLoss

@article{kim2023embeddistill,
    title={EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval},
    author={Kim, Seungyeon and Rawat, Ankit Singh and Zaheer, Manzil and Jayasumana, Sadeep and Sadhanala, Veeranjaneyulu and Jitkrittum, Wittawat and Menon, Aditya Krishna and Fergus, Rob and Kumar, Sanjiv},
    year={2023},
    eprint={2301.12005},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}