splade-co-condenser-marco trained on MS MARCO hard negatives with distillation

This is a SPLADE Sparse Encoder model finetuned from Luyu/co-condenser-marco on the msmarco-hard-negatives-cross-encoder-ms-marco-mini_lm-l-6-v2-scores dataset using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: Luyu/co-condenser-marco
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Training Dataset:
  • msmarco-hard-negatives-cross-encoder-ms-marco-mini_lm-l-6-v2-scores
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("arthurbresnu/splade-co-condenser-marco-msmarco-margin-mse-1-bs_128-lr_2e-05-lq_0.1-ld_0.08")
# Run inference
queries = [
    "skype how do i get video  voice",
]
documents = [
    'Speaker volume to adjust the sound. Slide the pointer up and down for volume, or select the speaker icon at the top of the volume control to mute your speaker. Select the Video button to add video to a Skype for Business call. Select the IM button to add instant messaging to a Skype for Business call.',
    'To make a free voice or video call on Skype for Web, you need to download a plugin. You can do this when you first sign in, or wait for when you want to make or receive your first call. Installing the plugin should only take a few moments, as the plugin is just 13.6MB. Making a voice or video call is simple.',
    'Century Dictionary and Cyclopedia. 1  n vogue The mode or fashion prevalent at any particular time; popular reception, repute, or estimation; common currency: now generally used in the phrase in vogue: as, a particular style of dress was then in. 2  n vogue General drift of ideas; rumor; report. ***.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[14.1007, 17.3336,  0.0000]])

Evaluation

Metrics

Sparse Information Retrieval

• Datasets: NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with SparseInformationRetrievalEvaluator

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ]
  }
  

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ]
  }
  

Training Details

Training Dataset

msmarco-hard-negatives-cross-encoder-ms-marco-mini_lm-l-6-v2-scores

• Dataset: msmarco-hard-negatives-cross-encoder-ms-marco-mini_lm-l-6-v2-scores
• Size: 522,751 training samples
• Columns: query, positive, negative, and label
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative	label
  type	string	string	string	float
  details	min: 4 tokensmean: 9.16 tokensmax: 48 tokens	min: 17 tokensmean: 80.22 tokensmax: 256 tokens	min: 20 tokensmean: 77.64 tokensmax: 234 tokens	min: -17.67mean: -1.3max: 7.3

• Samples:

  query	positive	negative	label
  can pasta be cooked and put in fridge	Cooked, refrigerated pasta is easily reheated by dropping it in boiling water for several seconds. Cooked pasta can also be frozen for up to 2 weeks. The pasta should be slightly cooled first and tossed with a bit of cooking or olive oil and placed into airtight freezer bags or containers.	CLOSE. 1 Cooked pasta can be stored in airtight containers in the refrigerator for 3 to 5 days. 2 Or, freeze cooked pasta for up to 2 weeks: Cool the pasta slightly, then drizzle with a little olive oil or cooking oil and toss gently. 3 Defrost a bag of frozen pasta in a colander in the sink by running tepid water over it.	-0.22228431701660156
  what is a oscillator circuit	An electronic oscillator is an electronic circuit that produces a periodic, oscillating electronic signal, often a sine wave or a square wave. Oscillators convert direct current (DC) from a power supply to an alternating current signal.	An electronic oscillator is an electronic circuit that produces a periodic, oscillating electronic signal, often a sine wave or a square wave. Oscillators convert direct current (DC) from a power supply to an alternating current (AC) signal. They are widely used in many electronic devices.	-0.22161579132080078
  what company makes the mclaren	Mclaren are a British company, the cars are manufactured in Surrey, England.	McLaren Automotive. McLaren Automotive (often simply McLaren) is a British automaker founded in 1963 by New Zealander Bruce McLaren and is based at the McLaren Technology Campus in Woking, Surrey. It produces and manufactures sports and luxury cars, usually produced in-house at designated production facilities.	-0.6905484199523926

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMarginMSELoss",
      "document_regularizer_weight": 0.08,
      "query_regularizer_weight": 0.1
  }
  

Evaluation Dataset

msmarco-hard-negatives-cross-encoder-ms-marco-mini_lm-l-6-v2-scores

• Dataset: msmarco-hard-negatives-cross-encoder-ms-marco-mini_lm-l-6-v2-scores
• Size: 10,000 evaluation samples
• Columns: query, positive, negative, and label
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative	label
  type	string	string	string	float
  details	min: 4 tokensmean: 8.99 tokensmax: 28 tokens	min: 23 tokensmean: 80.74 tokensmax: 229 tokens	min: 21 tokensmean: 78.11 tokensmax: 214 tokens	min: -17.67mean: -1.46max: 5.21

• Samples:

  query	positive	negative	label
  what is the legal meaning of a life estate	DEFINITION of 'Life Estate'. A type of estate that only lasts for the lifetime of the beneficiary. A life estate is a very restrictive type of estate that prevents the beneficiary from selling the property that produces the income before the beneficiary's death. But the estate cannot continue beyond the life of the beneficiary.	In common law and statutory law, a life estate is the ownership of land for the duration of a person's life. In legal terms it is an estate in real property that ends at death when ownership of the property may revert to the original owner, or it may pass to another person. The owner of a life estate is called a life tenant.	-1.0078916549682617
  what is the latest nvidia graphics card	GeForce GTX TITAN X is the ultimate graphics card. It combines the latest technologies and performance of the new NVIDIA Maxwell™ architecture to be the fastest, most advanced graphics card on the planet.	NVIDIA Titan X: The Ultimate Graphics Card Unleashed – Features Pascal GP102 GPU, 12 GB G5X Memory and a $1200 US Price. Today, NVIDIA launches their $1200 US, Titan X graphics card aiming the enthusiast and professional market. Do not mistake this card with last year’s GeForce GTX Titan X which featured the Maxwell GPU as the latest Titan X adopts the Pascal GPU architecture.	-0.10915374755859375
  what elements are used in a jewelry store	the most common metallic elements used in jewelry are gold, silver, platinum and copper. technically speaking, a diamond is an elemental form of carbon, if you want to includ … e that as well.most other jewelry components are compounds or mixtures of different elements. 1 person found this useful. Kimberly O'Brien. There are so many ways to sell your jewelry, but the problem is you pr…. 2 Shopping on Jeweler's Row Offers Maximum Selection and Minimum Prices Jeweler's Row is a stretch of several downtown Chicago blocks with an abundance of fine jewelry stores. 3 If there is a heaven for jewelry shoppers, this is it. 4 In one building….	Silver and gold are both malleable and lustrous, excellent properties for using these elements in jewelry. Metalloids share some characteristics of both metals and nonmetals. For example, silicon has luster and looks like a metal but does not conduct heat or electricity like a metal.	1.2301173210144043

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMarginMSELoss",
      "document_regularizer_weight": 0.08,
      "query_regularizer_weight": 0.1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 2e-05
• num_train_epochs: 35
• warmup_ratio: 0.05
• bf16: True
• load_best_model_at_end: True

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 35
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 2
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs