SentenceTransformer based on thebajajra/RexBERT-large

This is a sentence-transformers model finetuned from thebajajra/RexBERT-large on the nomic-embed-unsupervised-data dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: thebajajra/RexBERT-large
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • nomic-embed-unsupervised-data
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "Corners are still lifting.",
]
documents = [
    'Hello, I got my ender 3 a little over a year ago and have gotten many successful prints off of my machine. \n\nI have always had a problem with the corners of my prints lifting. I originally used a glass plate. That by itself was horrible, but then I added hairspray, and that worked. The problem was that on long prints, corners still lifted.\n\nAfter doing this for around 5 months I switched to a PEI sheet.\n\nThis worked comparably as well as the glass/hairspray combo, except the corners STILL LIFT on long prints.\n\nNow I have a PEI sheet on boro glass with an EZABL attached and the corners of my prints are STILL LIFTING.\n\nI don\'t know what i could possibly be doing wrong. The bed must be level. I get beautiful first layers, which I have tried to "smudge" around during printing and I can confirm that the plastic is being layed down solidly.\n\nIf anyone could enlighten me as to what is going on I would be thrilled.\n\nI do have my first layer printing at 30% speed with 150% layer width with the print cooling fan off as well. Printing PLA at 200C tool temp, 60C bed.',
    'These are awesome quart jars. They have a beautiful color, and I use them for storing soups, nuts and homemade nut milk. I would purchase them again.',
    'Great product. I purchased this item becuase my wrists would ache after triceps day at the gym. I would never be able to straighten my wrist and this helped in fixing that issue.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4861, 0.0704, 0.1127]])

Training Details

Training Dataset

nomic-embed-unsupervised-data

• Dataset: nomic-embed-unsupervised-data at 917bae6
• Size: 221,599,363 training samples
• Columns: query and document
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	min: 6 tokensmean: 34.17 tokensmax: 1024 tokens	min: 8 tokensmean: 166.07 tokensmax: 1024 tokens

• Samples:

  query	document
  Effect of steam reforming on methane-fueled chemical looping combustion with Cu-based oxygen carrier	Abstract The reduction characteristics of Cu-based oxygen carrier with H 2 , CO and CH 4 were investigated using a fixed bed reactor, TPR and TGA. Results showed that temperatures for the complete reduction of Cu-based oxygen carrier with H 2 and CO are 300 °C and 225 °C, respectively, while the corresponding temperature with CH 4 is 650 °C. The carbon deposition from CH 4 occurred at over 550 °C. CO-chemisorption experiments were also conducted on the oxygen carrier, and it was indicated that Cu-based oxygen carrier sinter seriously at 700 °C. In order to lower the required reduction temperature of oxygen carriers, a new chemical looping combustion (CLC) process with CH 4 steam reforming has been presented in this paper. The basic feasibility of the process was illustrated using CuO–SiO 2 . The new CLC process has the potential to replace the conventional gas-fired middle- and low-pressure steam and hot water boilers.
  who appointed onesicritus as chief pilot of the fleet	by the king to hold a conference with the Indian philosophers or Gymnosophists, the details of which have been transmitted to us from his own account of the interview. It was Onesicritus, whom Alexander first sent to summon Dandamis to his court. When later Onesicritus returned empty-handed with the reply of Dandamis, the King went to forest to visit Dandamis. When Alexander constructed his fleet on the Hydaspes, he appointed Onesicritus to the important position of pilot of the king's ship, or chief pilot of the fleet (). Onesicritus held this position not only during the descent of the Indus,
  when did the madonna of foligno go to paris	Madonna of Foligno hence the name. In 1799 it was carried to Paris, France by Napoleon. There, in 1802, the painting was transferred from panel to canvas by Hacquin and restored by Roser of Heidelberg. A note was made by the restorer: "Rapporto dei cittadini Guijon Vincent Tannay e Berthollet sul ristauro dei quadri di Raffaello conosciuto sotto il nome di Madonna di Foligno." In 1815, after the Battle of Waterloo, it was returned to Italy, where it was placed in the room with the Transfiguration in the Pinacoteca Vaticana of the Vatican Museum in the Vatican City. The painting is a "sacra

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

nomic-embed-unsupervised-data

• Dataset: nomic-embed-unsupervised-data at 917bae6
• Size: 1,113,579 evaluation samples
• Columns: query and document
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	min: 5 tokensmean: 31.98 tokensmax: 1024 tokens	min: 6 tokensmean: 161.48 tokensmax: 1024 tokens

• Samples:

  query	document
  Concise methods for the synthesis of chiral polyoxazolines and their application in asymmetric hydrosilylation	Seven polyoxazoline ligands were synthesized in high yield in a one-pot reaction by heating polycarboxylic acids or their esters and chiral β-amino alcohols under reflux with concomitant removal of water or the alcohol produced in the reaction. The method is much simpler and more efficient in comparison to those methods reported in the literature.The compounds were used as chiral ligands in the rhodium-catalyzed asymmetric hydrosilylation of aromatic ketones, and the effects of the linkers and the substituents present on the oxazoline rings on the yield and enantioselectivity investigated. Compound 2 was identified as the best ligand of this family for the hydrosilylation of aromatic ketones.
  On the road to a stronger public health workforce: visual tools to address complex challenges.	The Public Health Workforce Taxonomy: Revisions and Recommendations for Implementation
  140mm Jetflo fan availability?	I recently purchased a Nepton 280L, and would like to install an additional pair of 140mm Jetflo fans. Unfortunately they don't seem to be currently available, will they be in the future?Thank you so much!PS - I'm loving the cooling system!

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 192
• per_device_eval_batch_size: 128
• learning_rate: 1e-05
• num_train_epochs: 4
• warmup_steps: 1000
• bf16: True
• dataloader_num_workers: 20
• dataloader_prefetch_factor: 4
• ddp_find_unused_parameters: False

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 192
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 1000
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 20
• dataloader_prefetch_factor: 4
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu129
• Accelerate: 1.11.0
• Datasets: 4.3.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}