SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("dataera2013/midterm-ft-2")
# Run inference
sentences = [
    "QUESTION #2\\n...\\n\\nContext:\\nadults. - Impact on Self-Esteem Social media can trigger anxiety and depression. - Need for Mindful Usage Highlighting the importance of monitoring social media habits. - Complex Impacts Positive and negative effects on mental health require further exploration.  Questions - How can we balance the benefits of social media with its potential risks on mental health? - What role can regulations play in promoting healthier social media environments? - How can we measure the effectiveness of digital literacy programs in mitigating mental health issues? - What ethical considerations should be taken into account when leveraging AI for mental well-being on social media? - How can we ensure inclusivity and accessibility in peer support networks for\\n', additional_kwargs={}, response_metadata={})]",
    'adults. - Impact on Self-Esteem Social media can trigger anxiety and depression. - Need for Mindful Usage Highlighting the importance of monitoring social media habits. - Complex Impacts Positive and negative effects on mental health require further exploration.  Questions - How can we balance the benefits of social media with its potential risks on mental health? - What role can regulations play in promoting healthier social media environments? - How can we measure the effectiveness of digital literacy programs in mitigating mental health issues? - What ethical considerations should be taken into account when leveraging AI for mental well-being on social media? - How can we ensure inclusivity and accessibility in peer support networks for',
    'current encryption methods to quantum attacks, and how can they be addressed? 3. How can the balance between advancing encryption techniques and potential quantum vulnerabilities be maintained? 4. What strategies can organizations adopt to proactively address the risks posed by quantum computing in cryptography? 5. How can collaborative efforts between disciplines be optimized to drive innovation in quantum-safe encryption solutions?  Solutions 1. Adopt Post-Quantum Cryptography Embrace PQC to develop resilient cryptographic solutions against quantum attacks. 2. Enhance Encryption Techniques Focus on strengthening symmetric encryption algorithms and developing quantum-safe protocols. 3. Innovate Quantum-Safe Solutions Explore hybrid',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 238 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 238 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 7 tokensmean: 30.5 tokensmax: 181 tokens	min: 13 tokensmean: 126.02 tokensmax: 159 tokens

• Samples:

  sentence_0	sentence_1
  QUESTION #1\n	{"text": "What are the latest developments in quantum computing and how will they impact cryptography?", "response": "Opportunities 1. Post-Quantum Cryptography Active research in PQC offers robust cryptographic solutions resistant to quantum attacks. 2. Advancements in Encryption Strengthening symmetric encryption algorithms like AES-256 and developing quantum-safe protocols can enhance data security. 3. Innovative Quantum-Safe Encryption Hybrid encryption schemes combining classical and quantum-resistant algorithms can fortify data security. 4. Cross-Disciplinary Collaboration Collaborations across disciplines can lead to innovative approaches benefiting quantum computing and cryptography. 5. Enhanced Security Awareness Growing awareness
  QUESTION #2\n...\n\nContext:\n{"text": "What are the latest developments in quantum computing and how will they impact cryptography?", "response": "Opportunities	{"text": "What are the latest developments in quantum computing and how will they impact cryptography?", "response": "Opportunities 1. Post-Quantum Cryptography Active research in PQC offers robust cryptographic solutions resistant to quantum attacks. 2. Advancements in Encryption Strengthening symmetric encryption algorithms like AES-256 and developing quantum-safe protocols can enhance data security. 3. Innovative Quantum-Safe Encryption Hybrid encryption schemes combining classical and quantum-resistant algorithms can fortify data security. 4. Cross-Disciplinary Collaboration Collaborations across disciplines can lead to innovative approaches benefiting quantum computing and cryptography. 5. Enhanced Security Awareness Growing awareness
  Post-Quantum Cryptography Active research in PQC offers robust cryptographic solutions resistant to quantum attacks.	{"text": "What are the latest developments in quantum computing and how will they impact cryptography?", "response": "Opportunities 1. Post-Quantum Cryptography Active research in PQC offers robust cryptographic solutions resistant to quantum attacks. 2. Advancements in Encryption Strengthening symmetric encryption algorithms like AES-256 and developing quantum-safe protocols can enhance data security. 3. Innovative Quantum-Safe Encryption Hybrid encryption schemes combining classical and quantum-resistant algorithms can fortify data security. 4. Cross-Disciplinary Collaboration Collaborations across disciplines can lead to innovative approaches benefiting quantum computing and cryptography. 5. Enhanced Security Awareness Growing awareness

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• num_train_epochs: 5
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 5
• per_device_eval_batch_size: 5
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Framework Versions

• Python: 3.13.1
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}