SentenceTransformer based on google/bert_uncased_L-2_H-128_A-2

This is a sentence-transformers model finetuned from google/bert_uncased_L-2_H-128_A-2. It maps sentences & paragraphs to a 128-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/bert_uncased_L-2_H-128_A-2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 128, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("swardiantara/bert-tiny-yelp-k1-adaptive-euclidean")
# Run inference
sentences = [
    "What a great time!  This place was awesome!  I wish I lived right down the road, because I could see myself being a regular here.\\n\\nHonestly, these guys have got some talent, like real talent.  They can play the piano, sing, play guitar, and the drums.  Did I mention that they are all pretty hilarious, too.  I'd easily pay the $7 cover to get in here any day because it really is worth it.  \\n\\nI am sure I will be back in no time at all.  Next time, I am bringing a big group of friends.",
    "While we were out in Tempe, the husband and I were at a complete loss as to where we should eat.  I did a little yelping and sheepishly mentioned this place.  Now, we like country music and all, but I steer very clear of Toby Keith stuff.  I don't care for the in your face American Pride type stuff.  Anyway, we made out way to the restaurant, with entrances in front and back, and smelled the delicious bbq aroma floating in the air.\\n\\nThis place is HUGE!  I noticed that the capacity is just about 1000 people.  That is nuts.  there is a huge bar running down the middle and a ton of TV's spaced no more than 4 feet apart running all along the bar and all along the roof.  It's a good place to watch a game or the music videos.  I feel like I should also add that there is apparently a dress code for the servers.  All the girls are supposed to wear jingly black cowboy boots, super short denim shorts or skirts, skimpy tops that sometimes show too much skin on a less that perfectly built body, and a super sparkly rhinestoned out belt with a big ol buckle.\\n\\nAs far as the menu goes, the layout and look of it kind of suck.  I heard from our waitress that they are re-doing it, so let's hope that it will be for the better.  Almost everything on the menu actually looks like something I would order.  It's all pretty simple BBQ style stuff with some sandwiches and salads and some pricier entrees.  The appetizer menu looked delicious all around.  I laughed out loud when I read that there were deep fried twinkies on the dessert menu.  Classic.  I ordered the Oklahoma BBQ Beef sandwich with coleslaw as my side.  The sandwich was very filling and tasty.  The meat was juicy and the sauce was sweet and tangy.  The bun could have been more substantial but it was more than made up for with the yummy fried onion strings.  The coleslaw was just blah but I sampled the husband's potato salad and it was delicious!  Simple and tasty.\\n\\nOverall experience was pretty good.  Our waitress was nice and gave an honest opinion when we asked for one.  The atmosphere was kind of dull but that would probably be because we were there at such an odd time.  In case you were wondering, in the 30-45 minutes we were there, I counted 6 Toby Keith songs.  If I were into guns and in your face patriotism I would probably give this place 4 stars.",
    "First Watch on Black Canyon Hwy in Phoenix has amazing service and very good food! They're worth the drive. You won't be disappointed. Coincidentally, I went there b/c it was right next door to the Courtyard Marriott. I have been back several times and continue to receive excellent customer service and delicious food. \\n\\nMy favorite breakfast is their Tri-Athlete omelette; which I hesitatingly tried b/c it was a healthy choice but fell in love with it! Wonderfully roasted vegetables in an egg-white only omelette topped with salsa...muahhh, magnifico!!\\n\\nFor lunch I had their Pecan Dijon chicken salad...wow, another amazing dish full of flavors...good to the last drop!\\n\\nAnd lastly, their customer service...always SPOT ON! Once we had only 15 minutes. Informed the always-welcoming staff and VOILA, an excellent breakfast in exactly 15 minutes. I'll be back soon! \\n\\nThank you First Watch on Black Canyon Hwy!! You're awesome :)",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9970, 0.9999],
#         [0.9970, 1.0000, 0.9960],
#         [0.9999, 0.9960, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,925,010 training samples
• Columns: text_a, text_b, and label
• Approximate statistics based on the first 100 samples:

  	text_a	text_b	label
  type	string	string	list
  modality	text	text
  details	min: 6 tokensmean: 95.69 tokensmax: 128 tokens	min: 128 tokensmean: 128.0 tokensmax: 128 tokens	size: 2 elements

• Samples:

  text_a	text_b	label
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	First Watch on Black Canyon Hwy in Phoenix has amazing service and very good food! They're worth the drive. You won't be disappointed. Coincidentally, I went there b/c it was right next door to the Courtyard Marriott. I have been back several times and continue to receive excellent customer service and delicious food. \n\nMy favorite breakfast is their Tri-Athlete omelette; which I hesitatingly tried b/c it was a healthy choice but fell in love with it! Wonderfully roasted vegetables in an egg-white only omelette topped with salsa...muahhh, magnifico!!\n\nFor lunch I had their Pecan Dijon chicken salad...wow, another amazing dish full of flavors...good to the last drop!\n\nAnd lastly, their customer service...always SPOT ON! Once we had only 15 minutes. Informed the always-welcoming staff and VOILA, an excellent breakfast in exactly 15 minutes. I'll be back soon! \n\nThank you First Watch on Black Canyon Hwy!! You're awesome :)	[1.0, 0.0]
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	No, it's not worth it! Bellagio overcharged me and their accounting department never got back to me (I asked them for an itemized receipt 2 weeks ago). When I called them, I was (twice!) put on this music-less "dead" hold for 30-40 minutes. They're so full of themselves, they don't provide good customer service. Their wi-fi needed fixing when I stayed there. And by the way, the woman at the buffet was super rude to me. Finally, the famous waterworks never happened while I was standing outside in the cold waiting (though i asked the staff and they said they're up and running). No, don't waste your money! I won't and won't let my family and friends do that either.	[0.0, 1.0]
  dr. goldberg offers everything i look for in a general practitioner. he's nice and easy to talk to without being patronizing; he's always on time in seeing his patients; he's affiliated with a top-notch hospital (nyu) which my parents have explained to me is very important in case something happens and you need surgery; and you can get referrals to see specialists without having to see him first. really, what more do you need? i'm sitting here trying to think of any complaints i have about him, but i'm really drawing a blank.	Really, I should know better...\n\nIt was late at night, we didn't feel like fast food burger places, and I have driven past this place millions of times. \n\nIt's in a dark scary lot too close for comfort to Greenway square. Probably not the best place for a woman to go by herself late at night, but luckily I had the husband with me.\n\nAfter looking at the extensive menu, I ordered the chili relleno plate and my husband ordered a chicken burrito, beef taco, and a horchata. \n\nI couldn't finish my chili relleno plate. (Quick lesson in Mexican cooking- when preparing anything with chilis, its a good idea to remove the majority of the seeds. Not just because of the heat, but because its NOT fun to have a mouthful of seeds.) Besides the hundreds of chewy seeds in my mouth, the chili rellenos were lukewarm. The whole plate was pretty bland, and with every bite I took I knew my stomach would be paying for it later. So I only ate half, then popped 3 tums and hoped for the best. I ended not...	[0.0, 0.75]

• Loss: main.OrdinalProxyContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 10
• learning_rate: 2e-05
• load_best_model_at_end: True

All Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 10
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

• The bold row denotes the saved checkpoint.

Training Time

• Training: 2.2 hours
• Evaluation: 2.3 minutes
• Total: 2.3 hours

Framework Versions

• Python: 3.12.4
• Sentence Transformers: 5.5.1
• Transformers: 5.11.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.13.0
• Datasets: 2.21.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}