SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

License

The base model (paraphrase-multilingual-MiniLM-L12-v2) is Apache-2.0. However, this checkpoint was fine-tuned on 2ADT-Consulting/susu-parallel, which is dominated by Bible-derived text retrieved from YouVersion and subject to YouVersion's terms of service independently of any model license. Out of caution, and for consistency with our other Susu models trained on the same corpus (nllb-susu-v1/v2, both CC-BY-NC-4.0), we release these weights under CC-BY-NC-4.0 pending clearer guidance on whether fine-tuning on restricted-license text carries those restrictions to the resulting weights. Review the dataset's license notes before commercial use.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Les navires de Hiram, qui apportèrent de lor dOphir, amenèrent aussi dOphir une grande quantité de bois de sandal et des pierres précieuses.',
    'Xirami nun Sulemani xa walikɛe naxee fa xɛɛma ra kelife Ofiri bɔxi ma, nee man naxa fa wuri nun gɛmɛ tofanyie ra.',
    'Won a kolon won nun Ala na a ra, barima a bara a Xaxili fi won ma.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6592, 0.1617],
#         [0.6592, 1.0000, 0.0717],
#         [0.1617, 0.0717, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 225,516 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 13 tokensmean: 38.24 tokensmax: 93 tokens	min: 10 tokensmean: 50.44 tokensmax: 115 tokens

• Samples:

  anchor	positive
  He made it according to the instructions that Moses, the Lord 's servant, had given the Israelites, as it says in the Law of Moses: an altar made of stones which have not been cut with iron tools. On it they offered burnt sacrifices to the Lord , and they also presented their fellowship offerings.	alɔ Alatala xa konyi Munsa a fala Isirayilakae bɛ ki naxɛ. E naxa sɛrɛxɛbade gɛmɛ daaxi ti, wure mu nu din naxan na, alɔ a sɛbɛxi Tawureta Munsa kitaabui kui ki naxɛ. E naxa sɛrɛxɛ gan daaxie nun xanunteya sɛrɛxɛe ba Alatala bɛ na fari.
  Celle-ci ordonna à Hathac de rapporter sa réponse à Mardochée :.	Esita to na mɛ, a man naxa Hataki xɛɛ Morodekayi xɔn ma, a xa sa yi fala a bɛ.
  But there is something you do that is right—you hate the things that the Nicolaitans do. I also hate what they do.	«Kɔnɔ wo bara fe nde raba naxan fan. Wo bara Nikolasi xa ɲama xa fe xɔn, alɔ n fan a xɔnxi ki naxɛ.».

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• learning_rate: 2e-05
• num_train_epochs: 4
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 2.5 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}