SentenceTransformer

This is a sentence-transformers model trained on the train dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer

• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • train

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Head out on an adventure with this rollicking and energetic indie rock jam. With crunchy guitar riffs, a gutsy and passionate vocal performance, and a playful, brash, and slightly quirky delivery, its the perfect soundtrack for a summer road trip or day driving with friends. The powerful and strong beat will keep you amiable and good natured, while hanging out and searching for something new. Whether youre looking for some garage rock revival or indie pop vibes, this song will leave you feeling lively and ready for anything. So put on your best swaggering attitude and embrace the messy, alternative style of this indie rock anthem.',
    "Embark on a spirited journey with this vibrant and dynamic indie rock track. Featuring gritty guitar riffs, a bold and heartfelt vocal delivery, and a playful yet edgy eccentricity, it serves as an ideal companion for summer road trips or cruising with friends. With its driving, robust rhythm, this tune fosters a cheerful and easygoing vibe, perfect for hanging out and exploring new horizons. Whether you're drawn to garage rock revival or indie pop flavors, this anthem will energize you and ignite your adventurous spirit, inviting you to adopt a confident, rebellious flair.",
    'This album stands as a refined and intricate example of art rock, blending clever lyrics with a serene and tranquil ambiance. Its diverse fusion of pop rock, jazz rock, and prog rock evokes a fresh, spring-inspired mood that is simultaneously delicate and laid-back. Infused with whimsical, offbeat tones and psychedelic, intellectually challenging lyrics, it offers a theatrical and playful exploration of the Canterbury scene, making it essential listening for enthusiasts of album rock.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8889, 0.0683],
#         [0.8889, 1.0000, 0.1342],
#         [0.0683, 0.1342, 1.0000]])

Training Details

Training Dataset

train

• Dataset: train
• Size: 28,816 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 18 tokensmean: 100.88 tokensmax: 256 tokens	min: 35 tokensmean: 98.27 tokensmax: 235 tokens

• Samples:

  anchor	positive
  This song is a mix of psychedelic garage and art rock with elements of pop rock. It has a precious and whimsical quality with trippy and fractured lyrics. The vocals are both quirky and wistful, adding to the psychedelic pop vibe. The song draws inspiration from the British psychedelia scene, and incorporates eerie and bittersweet undertones. With elements of psychedelic folk and experimental rock, the track takes the listener on a journey through different musical landscapes, representing the evolving and experimental nature of the psychedelic genre.	Blending psychedelic garage and art rock with touches of pop rock, this track exudes a delicate, whimsical charm underscored by surreal and fragmented lyrics. The vocals, simultaneously quirky and nostalgic, enhance its psychedelic pop atmosphere. Influenced by the British psychedelia movement, the song weaves eerie and bittersweet tones, while its infusion of psychedelic folk and experimental rock guides the listener through a dynamic exploration of the genre's innovative and evolving soundscape.
  This song is an ethereal and hypnotic blend of ambient, post rock and drone with a touch of techno. The detached and laid back mellow vibe of the electronic beats is complemented by the soothing and gentle soundscapes. The atmospheric and spacious textures create a clinical and eerie feeling that is both experimental and minimal. This is an ambient techno track that will take you on a journey through dark ambient soundscapes and leave you feeling soothed and mesmerized.	This track offers a dreamy and entrancing fusion of ambient, post-rock, and drone, subtly infused with techno elements. Its relaxed, detached electronic rhythms are beautifully paired with calming, delicate soundscapes, crafting a spacious and atmospheric texture that feels both clinical and haunting. As an ambient techno piece, it guides listeners through shadowy ambient realms, leaving them both captivated and comforted.
  This High-energy pop rock song is a perfect blend of modern electric blues and regional blues. The confident and swaggering vocals will get your heart racing, while the blistering guitar solos and intense drumming will keep you on the edge of your seat. The passionate lyrics and earnest delivery are perfect for anyone who loves modern delta blues, juke joint blues, and electric delta blues. The sleazy, sexual undertones add just the right amount of brashness to this raucous and confrontational anthem.	This dynamic pop rock track masterfully fuses contemporary electric blues with authentic regional blues influences. Bold, swagger-filled vocals ignite your pulse, while searing guitar riffs and powerful drumming maintain a gripping intensity throughout. With heartfelt lyrics and a sincere performance, this raucous anthem—rich in sleazy, provocative undertones—is ideal for fans of modern delta blues, juke joint blues, and electric delta blues.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• learning_rate: 2e-05
• num_train_epochs: 8

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 8
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Framework Versions

• Python: 3.10.19
• Sentence Transformers: 5.1.2
• Transformers: 4.57.3
• PyTorch: 2.9.1
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}