distilbert-base-uncased finetuned on MSMARCO with MarginMSELoss + MultipleNegativesRankingLoss

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 300 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Dot Product
• Supported Modality: Text

• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kwondw/distilbert-base-uncased-msmarco-margin-mse-mnrl-100")
# Run inference
sentences = [
    'The weather is lovely today.',
    "It's so sunny outside!",
    'He drove to the stadium.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: NanoMSMARCO
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

Unnamed Dataset

• Size: 100 training samples
• Columns: anchor, positive, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, negative_7, negative_8, negative_9, negative_10, and label
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	negative_9	negative_10	label
  type	string	string	string	string	string	string	string	string	string	string	string	string	list
  details	min: 4 tokensmean: 8.99 tokensmax: 23 tokens	min: 31 tokensmean: 80.52 tokensmax: 177 tokens	min: 37 tokensmean: 83.34 tokensmax: 171 tokens	min: 14 tokensmean: 79.85 tokensmax: 240 tokens	min: 29 tokensmean: 76.91 tokensmax: 160 tokens	min: 29 tokensmean: 79.97 tokensmax: 198 tokens	min: 33 tokensmean: 81.91 tokensmax: 163 tokens	min: 14 tokensmean: 77.54 tokensmax: 214 tokens	min: 41 tokensmean: 79.54 tokensmax: 164 tokens	min: 22 tokensmean: 81.26 tokensmax: 181 tokens	min: 19 tokensmean: 79.04 tokensmax: 163 tokens	min: 25 tokensmean: 78.16 tokensmax: 155 tokens	size: 10 elements

• Samples:

  anchor	positive	negative_1	negative_2	negative_3	negative_4	negative_5	negative_6	negative_7	negative_8	negative_9	negative_10	label
  what county is santa rosa, ca	Sponsored Topics. Santa Rosa is the county seat of Sonoma County, California, United States. The 2010 census reported a population of 167,815. Santa Rosa is the largest city in California's Wine Country and fifth largest city in the San Francisco Bay Area, after San Jose, San Francisco, Oakland, and Fremont and 26th largest city in California.	Santa Rosa, CA is in the western part of California in the pacific United States. Closest County to Santa Rosa, CA: Sonoma County. Neighboring Cities to Santa Rosa, CA: Larkfield-Wikiup, Sebastopol, Roseland, Fulton. ZIP Codes for Santa Rosa, CA: 95405, 95404, 95407, 95401, 95403, 95409, 95472. 3-Digit ZIP Code Prefix for Santa Rosa, CA:	Closest Airport to Santa Rosa, CA. The closest airport to Santa Rosa is Charles M. Schulz–Sonoma County Airport (STS). Distance from Charles M. Schulz–Sonoma County Airport to Santa Rosa, CA is 11,5 kilometers (7,1 miles). Charles M. Schulz–Sonoma County Airport (STS) Charles M. Schulz–Sonoma County Airport is located approximately 4,3 kilometers (2,6 miles) south of Windsor and about 11,5 kilometers (7,1 miles) northwest of Santa Rosa. IATA airport code is STS.	South Santa Rosa County comprises the area from Holley and Navarre in the east to Gulf Breeze at the western end of the Gulf Breeze Peninsula (Fairpoint Peninsula), and along U.S. Highway 98.	Sonoma County comprises the Santa Rosa, CA Metropolitan Statistical Area, which is also included in the San Jose-San Francisco-Oakland, CA Combined Statistical Area. It is the northwesternmost county in the nine-county San Francisco Bay Area region.	The Santa Rosa Costco is one of the highest-grossing stores in the nation, partly because it draws significant revenue from Lake and Mendocino County. It is anticipated that the Ukiah store will largely redirect those customers, capturing the sales tax along with the sales.	Discover Santa Rosa. Just 55 miles north of San Francisco, in the urban heart of wine country, you'll find Santa Rosa: a city with a cool vibe and vibrant culture waiting to be explored. Santa Rosa is the happening hub for all there is to see, do, eat, drink, hike, bike. and explore in Sonoma County.	Sonoma County is a county in the U.S. state of California. As of the 2010 United States Census, its population was 483,878. Its county seat and largest city is Santa Rosa. It is located to the north of Marin County and the south of Mendocino County. It is west of Napa County and Lake County. Sonoma County comprises the Santa Rosa, CA Metropolitan Statistical Area, which is also included in the San Jose-San Francisco-Oakland, CA Combined Statistical Area.	The Location of the Santa Rosa Cemetery... The Santa Rosa Cemetery is located in Reeves County. The county seat for Reeves County is located in Pecos. Below are the GPS coordinates (latitude and longitude) that we are using for the location of the Santa Rosa Cemetery. Lat: 31.4240° (31° 25' 26) Lon: -103.4841° (-103° 29' 2)	Press Democrat Classifieds. Free and paid classified ads in Sonoma County, Santa Rosa, California. Browse classified ads. Post free classified ads.	In 1867, the county recognized Santa Rosa as an incorporated city and in 1868 the state officially confirmed the incorporation, making it officially the third incorporated city in Sonoma County, after Petaluma, incorporated in 1858, and Healdsburg, incorporated in 1867.	[1.187312126159668, 2.8033299446105957, 4.377563953399658, 1.0368633270263672, 5.135627746582031, ...]
  how to check a vacuum leak	There are many problems that vacuum leaks can cause with your car. Vacuum leaks can cause your car to have a fast idle speed and to hesitate, or even misfire, when accelerating. This guide will go over two ways to check for vacuum leaks on a car engine.	This video is pretty straight forward in it's approach to finding a vacuum leak. On thing I might mention is that not all vacuum leaks are at the engine. Some HVAC systems run on engine vacuum and can cause a leak under the dash in some cases.	Put some plumber's tape on the threads of the auto vent and screw it into position. Now you are ready to test the drainage. Let water run to check for leaks and then fill up the sink and drain it all at once to verify that it drains properly and that there are no leaks when a large amount of water is draining.	Vacuum leaks can be located with spray carburetor cleaner or a can of WD-40. If the area is obstructed by linkage or hoses, use an extension nozzle to pinpoint the area of the vacuum leak. If the engine speeds up when an area is sprayed, you are close to finding the leak. By John Bellah.	1 With the engine running, use a can of carburetor cleaner to spray down the hoses and vacuum connections. 2 If there's a change in the idle speed or if the engine runs smoother, you might have found the vacuum leak.	How to Find a Vacuum Leak. Air leaking into your car—not out of it—can be a problem. PM's Senior Automotive Editor, Mike Allen, outlines the tools and techniques proven to smoke out even the toughest to find vacuum leaks. As you've heard many times, your car's engine is nothing but a big air pump.	Then the pipe shall be inspected. This test is usually completed by the client when the system is put into service by the client. Section 14 Leak Testing 14-2 Piping/Mechanical Handbook 1996:Rev.2 Vacuum Leak Testing This is the hardest type of test with which to find a leak.	Step 2: Check for leaks. Be sure to check for leaks from the radiator drain plug, petcock, or the coolant temperature sensor. Step 3: Check to see if the check engine or low coolant level light is illuminated. If it is, turn off the engine and check the coolant level inside the reservoir.	Vacuum leaks can be located with spray carburetor cleaner or a can of WD-40. If the area is obstructed by linkage or hoses, use an extension nozzle to pinpoint the area of the vacuum leak.	Steps. 1 1. Safety first - If the system you are looking at has something that burns in (like natural gas or propane), make certain that the area is well ventilated and there is nothing that will make a spark while you do your test. 2 2. Mix a few tablespoons of dish soap into a container of water. 3. Pressurize the system (turn on the gas, or use an air supply of some kind) 4. Wet down the suspected parts with the soap solution. If you see bubbles form on them, then you have found the leak! This approach will work on tires, inner tubes, gas lines pretty much anything with a gas under pressure inside.	Remove Blockages. If the Dyson vacuum has lost suction because of a blockage, removal is the answer. First, separate the hose from the Dyson vacuum and the vacuum wand. If the hose has a plastic U-bend attached, this is where blockages tend to build up.Check the U-bend for clogs.pply a few dots of super glue to the plastic ring and slide the hose end in place. Press together until the glue holds. Allow the ring to dry for several hours. Connect the vacuum hose to the outer plastic cuff. Attach this to the vacuum and the vacuum wand.	[2.5488624572753906, 4.985450476408005, -3.698561668395996, -1.9865045547485352, -0.3907012939453125, ...]
  most expensive cities in the world	Singapore has retained its position as the world's most expensive city, according to research by the Economist Intelligence Unit (EIU). The top five most expensive cities in the world remain unchanged from a year earlier and include, in descending order, Paris, Oslo, Zurich and Sydney.	DUBAI // Dubai has been ranked the world’s second-most expensive city in which to stay, and industry experts say tourists are travelling to nearby emirates to avoid the rising costs of hotels. In a list compiled by Bloomberg, Geneva was the most expensive city, and Kuwait and Zurich were third and fourth.	March 7, 2017. Once again, international design and consultancy firm Arcadis has named New York City as the world's most expensive city in which to build, according to its annual International Construction Costs Index.	New York was the world’s most expensive city including rent as of April 2015, according to a new analysis by UBS. KENA BETANCUR/AFP/GETTY IMAGES. New York City takes the top spot in the annual ranking of the world’s most expensive cities, according to an analysis by UBS. The cost of goods and services was higher in just two other cities— Zurich and Geneva— but they were less expensive than New York after including rent.	The 20 most expensive cities in the world to live in revealed: Hamilton in Bermuda tops the rankings with San Francisco and New York close behind. 1 Hamilton, Bermuda, has been revealed as the most expensive city in the world. 2 20 cities with the highest cost of living include New York, Sydney and London.	The world's most expensive city is …. When one thinks of the world's most expensive cities, images of New York and London probably pop to mind. But those two do not even make the list of the 10 most expensive cities in which to live. Instead, it's dominated by Asian and African cities, as well as two Swiss ones — Zurich and Geneva — according to Mercer's annual 2016 Cost of Living Rankings.	The Most Expensive Buildings in the World. 10. Antilla, Mumbai, India – $2.53 billion. 9. City of Dreams, Macau, China – $2.75 billion. 8. The Venetian Macau, Macau, China – $2.97 billion. 7. Wynn Resort, Las Vegas – $3.26 billion. 6. One World Trade Center, New York – $3.8 billion. 5. The Shard, London, England – $3.9 billion.	London, United Kingdom. Having been rated as the world’s most expensive city to visit for 24 hours it should come as no surprise that London would be on this list somewhere. What may be surprising to some of you, however, is that there are at least 24 cities even more expensive. Brisbane, Australia.	London, United Kingdom. Having been rated as the world’s most expensive city to visit for 24 hours it should come as no surprise that London would be on this list somewhere. What may be surprising to some of you, however, is that there are at least 24 cities even more expensive. 24. Brisbane, Australia. Although it is not Australia’s most expensive city it does arguably have one of the world’s most expensive public transportation systems (trailing only London and Oslo) 23. Canberra, Australia. Moving from one Australian city to another this time we’re in the capital of the land down under.	The 10 most expensive cities to live in around the world in 2017. LONDON — The most expensive cities to live in around the world in 2017 have been revealed in the 13th Annual Demographia International Housing Affordability Survey: 2017.	The 10 Most Expensive Cities In The World. Tokyo is back on top as the most expensive city in the world, according to new data from the Economist Intelligence Unit. In the company's new worldwide cost of living survey, the Japanese city has outpaced last year's winner Zurich for the undesirable ranking as the world's priciest city.	[0.753758430480957, 0.9305486679077148, -0.0030488967895507812, 0.40877246856689453, 0.7249746322631836, ...]

• Loss: main.CombinedMarginMSEMNRLLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Training Time

• Training: 12.6 seconds
• Evaluation: 12.9 seconds
• Total: 25.5 seconds

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}