SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '',
    '# Copying, moving, and renaming objects\n\nThe CopyObject operation creates a copy of an object that\'s already stored in Amazon S3.\n\nYou can create a copy of an object up to 5 GB in a single atomic operation. However, to copy an object that\'s larger than 5 GB, you must use a multipart upload using the AWS CLI or AWS SDKs. For more information, see Copying an object using multipart upload.\n\n###### Note\n\nTo maintain the performance benefit of an object you uploaded using multipart upload, you must copy the object using multipart upload using the AWS CLI or AWS SDK instead of the S3 console. For more information, see Copying an object using multipart upload.\n\nUsing the CopyObject operation, you can:\n\n- Create additional copies of objects.\n- Rename objects by copying them and deleting the original ones.\n- Copy or move objects from one bucket to another, including across AWS Regions (for example, from us-west-1 to eu-west-2). When you move an object, Amazon S3 copies the object to the specified destination and then deletes the source object. NoteCopying or moving objects across AWS Regions incurs bandwidth charges. For more information, see Amazon S3 Pricing.\n- Change object metadata. Each Amazon S3 object has metadata. This metadata is a set of name-value pairs. You can set object metadata at the time you upload an object. After you upload the object, you can\'t modify the object metadata. The only way to modify object metadata is to make a copy of the object and set the metadata. To do so, in the copy operation, set the same object as the source and target.  Some object metadata is system metadata and other is user-defined. You can control some of the system metadata. For example, you can control the storage class and the type of server-side encryption to use for the object. When you copy an object, user-controlled system metadata and user-defined metadata are also copied. Amazon S3 resets the system-controlled metadata. For example, when you copy an object, Amazon S3 resets the creation date of the copied object. You don\'t need to set any of these system-controlled metadata values in your copy request.  When copying an object, you might decide to update some of the metadata values. For example, if your source object is configured to use S3 Standard storage, you might choose to use S3 Intelligent-Tiering for the object copy. You might also decide to alter some of the user-defined metadata values present on the source object. If you choose to update any of the object\'s user-configurable metadata (system or user-defined) during the copy, then you must explicitly specify all of the user-configurable metadata present on the source object in your request, even if you are changing only one of the metadata values. NoteWhen copying an object by using the Amazon S3 console, you might receive the error message "Copied metadata can\'t be verified." The console uses headers to retrieve and set metadata for your object. If your network or browser configuration modifies your network requests, this behavior might cause unintended metadata (such as modified Cache-Control headers) to be written to your copied object. Amazon S3 can\'t verify this unintended metadata.To address this issue, check your network and browser configuration to make sure it doesn\'t modify headers, such as Cache-Control. For more information, see The Shared Responsibility Model. For more information about object metadata, see Working with object metadata.\n\n###### Copying archived and restored objects',
    "The request with HTTP 1.0 and omitting the Host header is as follows:\n\n```\nGET /example.com/homepage.html HTTP/1.0\n```\n\nFor information about DNS-compatible names, see Limitations. For more information about keys, see Keys.\n\n###### Example â\x80\x93 Virtual-hostedâ\x80\x93style URLs and requests\n\nThis example uses the following:\n\n- Bucket name â\x80\x90 amzn-s3-demo-bucket1\n- Region â\x80\x90 Europe (Ireland)\n- Key name â\x80\x90 homepage.html\n\nThe URL is as follows:\n\n```\nhttp://amzn-s3-demo-bucket1.s3.eu-west-1.amazonaws.com/homepage.html\n```\n\nThe request is as follows:\n\n```\nGET /homepage.html HTTP/1.1 Host: amzn-s3-demo-bucket1.s3.eu-west-1.amazonaws.com\n```\n\n###### Example â\x80\x93 CNAME alias method\n\nTo use this method, you must configure your DNS name as a CNAME alias for bucket-name.s3.us-east-1.amazonaws.com. For more information, see Customizing Amazon S3 URLs with CNAME records.\n\nThis example uses the following:\n\n- Bucket Name â\x80\x90 example.com\n- Key name â\x80\x90 homepage.html\n\nThe URL is as follows:\n\n```\nhttp://www.example.com/homepage.html\n```\n\nThe example is as follows:\n\n```\nGET /homepage.html HTTP/1.1 Host: www.example.com\n```\n\n## Customizing Amazon S3 URLs with CNAME records\n\nDepending on your needs, you might not want s3.region-code.amazonaws.com to appear on your website or service. For example, if you're hosting website images on Amazon S3, you might prefer http://images.example.com/ instead of http://images.example.com.s3.us-east-1.amazonaws.com/. Any bucket with a DNS-compatible name can be referenced as follows:  http://BucketName.s3.Region.amazonaws.com/[Filename], for example, http://images.example.com.s3.us-east-1.amazonaws.com/mydog.jpg. By using CNAME, you can map images.example.com to an Amazon S3 hostname so that the previous URL could become http://images.example.com/mydog.jpg.\n\nYour bucket name must be the same as the CNAME. For example, if you create a CNAME to map images.example.com to images.example.com.s3.us-east-1.amazonaws.com, both http://images.example.com/filename and http://images.example.com.s3.us-east-1.amazonaws.com/filename will be the same.\n\nThe CNAME DNS record should alias your domain name to the appropriate virtual hostedâ\x80\x93style hostname. For example, if your bucket name and domain name are images.example.com and your bucket is in the US East (N. Virginia) Region, the CNAME record should alias to images.example.com.s3.us-east-1.amazonaws.com.\n\n```\nimages.example.com CNAME images.example.com.s3.us-east-1.amazonaws.com.\n```\n\nAmazon S3 uses the hostname to determine the bucket name. So the CNAME and the bucket name must be the same. For example, suppose that you have configured www.example.com as a CNAME for www.example.com.s3.us-east-1.amazonaws.com. When you access http://www.example.com, Amazon S3 receives a request similar to the following:\n\n```\nGET / HTTP/1.1 Host: www.example.com Date: date Authorization: signatureValue\n```",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 194 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 194 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 2 tokensmean: 22.15 tokensmax: 54 tokens	min: 5 tokensmean: 470.86 tokensmax: 512 tokens

• Samples:

  sentence_0	sentence_1
  QUESTION #1	```
  QUESTION #2	```
  QUESTION #1	```

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.7.0
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}