SPLADE Sparse Encoder

This is a SPLADE Sparse Encoder model finetuned from naver/splade-v3 using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

This model was made as part of an Advanced Information Retrieval Course in the scope of our project which is meant to recommend games based on their description. See also our github: https://github.com/Spivonxe/There-are-no-games

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: naver/splade-v3
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("mazombieme/There-Are-No-Games")
# Run inference
queries = [
    "artifacts powerful",
]
documents = [
    'Dreamscaper By night, delve deep into your subconscious and discover powerful artifacts to conquer your nightmares. By day, explore the city of Redhaven and build relationships to unlock the power of your dreams. DREAM. DIE. WAKE. REPEAT.',
    "The Mystery of Devils House The Mystery of Devil's House is a 2D Horror Action Platformer based on the story of the Haunted house called William's House located in Massachusetts investigated by The Ghostbuster Crew: International.",
    'KoroNeko Roll your way through a cozy, kawaii world full of charming characters and challenging puzzles to save your siblings from Strawberry the Witch!',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[12.4569,  0.0118,  0.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 39,381 training samples
• Columns: query and document
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	min: 4 tokensmean: 7.17 tokensmax: 22 tokens	min: 10 tokensmean: 50.35 tokensmax: 103 tokens

• Samples:

  query	document
  control difficulties	Manabi SandBox This is an action game in which you control a knight and explore unknown ruins. The goal is to collect all the fruits on the map while exterminating monsters. Overcome various difficulties with simple actions and ingenuity. The Knights never die, so don't give up and solve all the riddles.
  atmospheric every mystery narrativedriven mansion tension	Crimson Mansion Uncover the secrets of a forgotten mansion where every room raises new questions. A narrative-driven exploration game full of mystery and atmospheric tension.
  visually games edutainment pointandclick puzzle adventure	Endacopia Endacopia is a Point-and-Click Puzzle Adventure that is visually reminiscent of early computer edutainment games... with underlying horrors.

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
      "document_regularizer_weight": 0.003,
      "query_regularizer_weight": 0
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 4,913 evaluation samples
• Columns: query and document
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	min: 4 tokensmean: 6.97 tokensmax: 16 tokens	min: 6 tokensmean: 49.39 tokensmax: 118 tokens

• Samples:

  query	document
  cones playground park explore	Pipo Park Pipo Park is a messy and happy digital playground. Explore a small park, play football, throw traffic cones, balls, dogs, and more... Have a nice time!
  sentient united employee dark	Quantum Eye Set in a dark, futuristic united states, all citizens are watched under a highly-advanced supercomputer known as the "Quantum Eye" Machine. You are an employee at the Quantum Eye facility, but quantum eye goes sentient. Your goal is simple, SHUT IT DOWN...
  five solve in twisted 2 powerful	Hanaja's Body 2 in One Indulge in this twisted tale of transformations as five characters struggle for control of one body: yours! Mix and match between five different, uniquely powerful forms to defeat dangerous foes and solve puzzles to escape the tome you've been trapped in before it's too late.

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
      "document_regularizer_weight": 0.003,
      "query_regularizer_weight": 0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates
• router_mapping: {'query': 'query', 'answer': 'document'}

All Hyperparameters

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {'query': 'query', 'answer': 'document'}
• learning_rate_mapping: {}

Training Logs

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.2.0
• Transformers: 4.57.1
• PyTorch: 2.9.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}