SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("along26/all-MiniLM-L6-v2_multilingual_malaysian-v4")
# Run inference
sentences = [
    'How can we design small molecule inhibitors of viral protein targets to prevent the replication of the influenza virus?',
    'Bagaimanakah kita boleh mereka bentuk perencat molekul kecil sasaran protein virus untuk mencegah replikasi virus influenza?',
    "How does the Malaysian government's authoritarian approach to dissent and free speech stifle progressive movements and limit the potential for democratic reform?",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3951, 0.3134],
#         [0.3951, 1.0000, 0.3861],
#         [0.3134, 0.3861, 1.0000]])

Evaluation

Metrics

Triplet

• Evaluated with TripletEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 415,570 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 15 tokensmean: 230.67 tokensmax: 512 tokens	min: 19 tokensmean: 273.54 tokensmax: 512 tokens	min: 14 tokensmean: 239.81 tokensmax: 512 tokens

• Samples:

  anchor	positive	negative
  How has the culture of corruption and cronyism under Najib Razak's administration affected the Malaysian economy and social fabric?	Bagaimanakah budaya rasuah dan kronisme di bawah pentadbiran Najib Razak menjejaskan ekonomi dan fabrik sosial Malaysia?	What is the role of the pancreas in the human digestive system and how does its anatomy support this function?
  Why have some opposition politicians in Malaysia criticized the government's handling of the 1MDB scandal and called for more transparency?	Mengapa beberapa ahli politik pembangkang di Malaysia mengkritik pengendalian kerajaan terhadap skandal 1MDB dan meminta lebih ketelusan?	The formation of heavy elements (nucleosynthesis) inside a star involves several processes, which are affected by the star's evolution. These processes include:1. Hydrogen burning (nuclear fusion): This is the initial stage of a star's life, where hydrogen nuclei (protons) combine to form helium nuclei (alpha particles) through a series of reactions called the proton-proton chain or the CNO cycle (carbon-nitrogen-oxygen). This process releases a large amount of energy in the form of light and heat, which causes the star to shine.2. Helium burning (triple-alpha process): As the hydrogen in the core of the star is depleted, the core contracts and heats up, initiating the fusion of helium nuclei into heavier elements like carbon and oxygen. This process involves the combination of three helium nuclei (alpha particles) to form a carbon nucleus.3. Carbon burning: In more massive stars, the core temperature increases further, allowing carbon nuclei to fuse with helium nuclei to form ox...
  How has Najib Razak's corruption allegedly contributed to social inequality and poverty in Malaysia?	Bagaimanakah rasuah Najib Razak didakwa menyumbang kepada ketidaksamaan sosial dan kemiskinan di Malaysia?	To estimate the age of a supermassive black hole with a mass of 1 billion solar masses, we can assume that it formed shortly after the Big Bang. The age of the universe is approximately 13.8 billion years old, so the black hole's age would be close to this value.Now, let's discuss the theoretical process for the formation and evolution of supermassive black holes in the early universe based on current astrophysical models.1. Direct collapse: In the early universe, some regions with high-density gas could have collapsed directly into black holes without forming stars first. These black holes, called "seed" black holes, could then grow by accreting mass from their surroundings. This process is more efficient in the early universe due to the higher density of gas and the absence of supernova explosions, which can disperse gas and hinder black hole growth.2. Stellar remnants: Massive stars in the early universe could have collapsed into black holes after their lifetimes. These black ...

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 12 tokensmean: 219.95 tokensmax: 512 tokens	min: 17 tokensmean: 263.51 tokensmax: 512 tokens	min: 15 tokensmean: 236.31 tokensmax: 512 tokens

• Samples:

  anchor	positive	negative
  Consider a graph with 8 vertices and 12 edges. Determine if the graph contains a perfect matching. If it does, provide one example of a perfect matching. If it does not, explain why a perfect matching is not possible.	Pertimbangkan graf dengan 8 bucu dan 12 tepi. Tentukan sama ada graf mengandungi padanan sempurna. Jika ya, berikan satu contoh padanan yang sempurna. Jika tidak, jelaskan mengapa padanan yang sempurna tidak dapat dilakukan.	The 1MDB scandal and the corruption charges against former Malaysian Prime Minister Najib Razak offer several important lessons for Malaysia and other countries:1. Stronger checks and balances: The 1MDB scandal highlighted the need for stronger checks and balances in government agencies and institutions. Malaysia should consider implementing additional measures to prevent the misuse of power and public funds, including more robust auditing and oversight mechanisms, as well as stronger whistleblower protections.2. Transparency and accountability: The lack of transparency and accountability surrounding 1MDB contributed to the scandal. Malaysia should prioritize transparency in government operations, including procurement processes and financial transactions. Implementing measures such as open data initiatives and requiring greater disclosure from government-linked companies could help promote accountability and reduce opportunities for corruption.3. Strengthening law enforcement: The...
  What is the probability of flipping a fair coin three times and getting exactly two heads in a row?	Apakah kebarangkalian membalikkan syiling saksama tiga kali dan mendapat tepat dua kepala berturut-turut?	Why is corruption so rampant in Malaysia, with politicians and government officials often caught engaging in unethical practices?
  Why have there been allegations of corruption and mismanagement in Malaysia's state-owned enterprises, and what measures have been taken to address these issues?	Mengapa terdapat tuduhan rasuah dan salah urus dalam perusahaan milik kerajaan Malaysia, dan apakah langkah-langkah yang telah diambil untuk menangani isu-isu ini?	What is the pKa value of acetic acid, and how does it affect its acid strength when compared to other organic acids such as citric acid or benzoic acid? Provide an explanation for your answer using acid-base reaction principles in organic chemistry.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• weight_decay: 0.01
• warmup_steps: 100
• fp16: True
• load_best_model_at_end: True

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 100
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 4.0.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}