SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("dinushiTJ/nz-research-commons-embedding-gemma-v2")
# Run inference
queries = [
    'maori_origin',
]
documents = [
    'title: Understanding cultural relationships: Whānau, whanaungatanga and Māori student attainment of university entrance in a mainstream secondary school in Aotearoa, New Zealand\n\nauthors: McGill, Kristin\n\nsubjects: Cultural relationships\n\nabstract: This study seeks to understand the importance of cultural relationships in supporting Māori student achievement of University Entrance.  This research is based on the stories of five female ākonga Māori, all of whom completed five years of secondary education, and their whānau.  It looks deeply into their relational experiences of whanaungatanga and whānautanga with their school, and the impact this had on their academic achievement of NCEA Level 3 and University Entrance. \r\n\r\nThe results highlight the importance of culturally grounded transformative praxis and the risk of attempting to incorporate culturally located principles such as whānau and whanaungatanga into a schooling context, while still operating within historical hegemonic frameworks.\n\ntext: Understanding cultural relationships: Whānau, whanaungatanga and Māori student attainment of university entrance in a mainstream secondary school in Aotearoa, New Zealand This study seeks to understand the importance of cultural relationships in supporting Māori student achievement of University Entrance.  This research is based on the stories of five female ākonga Māori, all of whom completed five years of secondary education, and their whānau.  It looks deeply into their relational experiences of whanaungatanga and whānautanga with their school, and the impact this had on their academic achievement of NCEA Level 3 and University Entrance. \r\n\r\nThe results highlight the importance of culturally grounded transformative praxis and the risk of attempting to incorporate culturally located principles such as whānau and whanaungatanga into a schooling context, while still operating within historical hegemonic frameworks.\n\nyear: 2023',
    'title: Editorial: Pacific education: research and practice.\n\nauthors: Strachan, Jane\n\nsubjects: educational anthropology\n\nabstract: The article discusses various reports published within the issue including one by Tanya Wendt Samu on the call for teachers to be responsive to the diversities between group of learners as well as within groups of learners and another by Fran Cahill on the discussion of the difficulties that Samoan adolescents have in living within the traditional Samoan culture of home.\n\ntext: Editorial: Pacific education: research and practice. The article discusses various reports published within the issue including one by Tanya Wendt Samu on the call for teachers to be responsive to the diversities between group of learners as well as within groups of learners and another by Fran Cahill on the discussion of the difficulties that Samoan adolescents have in living within the traditional Samoan culture of home.\n\nyear: 2006',
    'title: An evaluation of Te Rau Puawai workforce 100: Perspectives of Te Rau Puawai bursars\n\nauthors: Nikora, Linda Waimarie\n\nsubjects: Maori students\n\nabstract: The Te Rau Puawai programme is an attempt to change the nature of the Maori\r\nmental health workforce. To do this, Maori with aspirations to work, or to continue to\r\nwork in the mental health workforce, are supported, financially and academically, to\r\ncomplete a tertiary qualification relevant to the field.\r\nTo evaluate the Te Rau Puawai programme, the Ministry of Health commissioned the\r\nMaori and Psychology Research Unit of the University of Waikato in July 2001. The\r\noverall aim of the evaluation was to provide the Ministry with a clearer understanding\r\nof the programme including: the perceived critical success factors, the barriers if any\r\nregarding Te Rau Puawai, the impact of the programme, the extent to which the\r\nprogramme may be transferable, gaps in the programme, and suggested\r\nimprovements.\r\nThe evaluation team set out to gather the experiences and perspectives of recipients of\r\nTe Rau Puawai services by asking all bursars to complete a questionnaire and\r\nvolunteer for follow up interviews or focus groups. Sixty two bursars responded to\r\nour questionnaire, and we complete focus group or individual follow up interviews\r\nwith 19 bursars.\n\ntext: An evaluation of Te Rau Puawai workforce 100: Perspectives of Te Rau Puawai bursars The Te Rau Puawai programme is an attempt to change the nature of the Maori\r\nmental health workforce. To do this, Maori with aspirations to work, or to continue to\r\nwork in the mental health workforce, are supported, financially and academically, to\r\ncomplete a tertiary qualification relevant to the field.\r\nTo evaluate the Te Rau Puawai programme, the Ministry of Health commissioned the\r\nMaori and Psychology Research Unit of the University of Waikato in July 2001. The\r\noverall aim of the evaluation was to provide the Ministry with a clearer understanding\r\nof the programme including: the perceived critical success factors, the barriers if any\r\nregarding Te Rau Puawai, the impact of the programme, the extent to which the\r\nprogramme may be transferable, gaps in the programme, and suggested\r\nimprovements.\r\nThe evaluation team set out to gather the experiences and perspectives of recipients of\r\nTe Rau Puawai services by asking all bursars to complete a questionnaire and\r\nvolunteer for follow up interviews or focus groups. Sixty two bursars responded to\r\nour questionnaire, and we complete focus group or individual follow up interviews\r\nwith 19 bursars.\n\nyear: 2002-05-01',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9017, -0.1908,  0.8361]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 5,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 6 tokensmean: 6.75 tokensmax: 8 tokens	min: 55 tokensmean: 702.62 tokensmax: 1882 tokens	min: 93 tokensmean: 651.33 tokensmax: 2048 tokens

• Samples:

  anchor	positive	negative
  maori_origin	title: Biteauthors: Mitchell, Torisubjects: poetryabstract: Bite is a collection of blunt confessional poetry draped in honesty, identity, and 21st century feminism, examined through lived experience with eating disorders, sexuality, trauma, disconnect from family and culture, love and loss, and what it means to be a girl growing up. The collection explores these raw concepts with vulnerability, through the eyes of ‘wolf girl’, in hopes that talking loudly about these topics will force the public to challenge the silence and stigma around them.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.3
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• prompts: task: classification | query:

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: task: classification | query:
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 1.1 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.7.0
• Transformers: 4.57.0.dev0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}