SentenceTransformer based on microsoft/harrier-oss-v1-270m

This is a sentence-transformers model finetuned from microsoft/harrier-oss-v1-270m. It maps sentences & paragraphs to a 640-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: microsoft/harrier-oss-v1-270m
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 640 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 640, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Instruct: Represent this Buddhist passage for semantic retrieval.\nQuery: Bhagavā taṃ paṭiggahetvā ānandattheraṃ āṇāpesi – ‘‘ imaṃ phalaṃ madditvā pānaṃ dehī ’’ ti. Thero tathā akāsi. Bhagavā ambarasaṃ pivitvā ambaṭṭhiṃ uyyānapālassa datvā ‘‘ imaṃ ropehī ’’ ti āha. So vālukaṃ viyūhitvā taṃ ropesi, ānandatthero kuṇḍikāya udakaṃ āsiñci.',
    'The Blessed One accepted it and instructed Venerable Ānanda: “Crush this fruit and give it as a drink.” The Elder did so. The Blessed One, having drunk the mango juice, gave the mango seed to the gardener and said, “Plant this.” He dug up the sand and planted it, and Venerable Ānanda poured water from a vessel.',
    '“The foremost of my disciples, monks, who are very learned. Who are mindful. Who are clear-headed. Who are resolute. Who are attendants, is Ānanda” –',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 640]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.8314, 0.1940],
#         [0.8314, 1.0000, 0.2902],
#         [0.1940, 0.2902, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 744,489 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 22 tokensmean: 90.93 tokensmax: 128 tokens	min: 9 tokensmean: 55.09 tokensmax: 128 tokens

• Samples:

  anchor	positive
  Instruct: Represent this Buddhist passage for semantic retrieval.Query: Namo tassa bhagavato arahato sammāsambuddhassa	Homage to the Blessed One, the Perfected One, the Fully Self-Awakened Buddha
  Instruct: Represent this Buddhist passage for semantic retrieval.Query: 1. Evaṃ me sutaṃ – ekaṃ samayaṃ bhagavā sāvatthiyaṃ viharati jetavane anāthapiṇḍikassa ārāme. Tatra kho bhagavā bhikkhū āmantesi – ‘‘ bhikkhavo ’’ ti. ‘‘ Bhadante ’’ ti te bhikkhū bhagavato paccassosuṃ. Bhagavā etadavoca –	Thus have I heard — on one occasion the Blessed One was dwelling at Sāvatthī in Jeta's Grove, in Anāthapiṇḍika’s Park. There the Blessed One addressed the monks: “Monks!” “Venerable sir,” those monks replied to the Blessed One. The Blessed One said:
  Instruct: Represent this Buddhist passage for semantic retrieval.Query: 2.‘‘ Nāhaṃ, bhikkhave, aññaṃ ekasaddampi samanupassāmi yaṃ evaṃ purisassa cittaṃ pariyādāya tiṭṭhati yathayidaṃ, bhikkhave, itthisaddo. Itthisaddo, bhikkhave, purisassa cittaṃ pariyādāya tiṭṭhatī ’’ ti. Dutiyaṃ.	“Monks, I do not see a single sound that invades his mind and remains in a person as much as the sound of a woman. The sound of a woman, monks, overpowers a man’s mind.” The second.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 15,180 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 21 tokensmean: 82.69 tokensmax: 128 tokens	min: 7 tokensmean: 58.92 tokensmax: 128 tokens

• Samples:

  anchor	positive
  Instruct: Represent this Buddhist passage for semantic retrieval.Query: ‘‘ Nāhaṃ, bhikkhave, aññaṃ ekarūpampi samanupassāmi yaṃ evaṃ purisassa cittaṃ pariyādāya tiṭṭhati yathayidaṃ, bhikkhave, itthirūpaṃ. Itthirūpaṃ, bhikkhave, purisassa cittaṃ pariyādāya tiṭṭhatī ’’ ti. Paṭhamaṃ.	“Monks, I do not see a single sight that invades his mind and remains in a person as much as the sight of a woman. The sight of a woman, monks, overpowers a man’s mind.” The first.
  Instruct: Represent this Buddhist passage for semantic retrieval.Query: Paṇihitaacchavaggo pañcamo.	The Chapter on the Directed and Clear is the fifth.
  Instruct: Represent this Buddhist passage for semantic retrieval.Query: 8. Kalyāṇamittādivaggo	8. Chapter on Good Friendship

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• gradient_accumulation_steps: 2
• learning_rate: 2e-05
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_steps: 0.05
• fp16: True
• remove_unused_columns: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.05
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: False
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 6.1 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}