Sparseup-fr-v0

Description

This is a sparse encoder model trained on the lightonai/embeddings-fine-tuning-filtered-fr dataset using the sentence-transformers library. It maps inputs to a 50240-dimensional sparse vector space and can be used for semantic search and sparse retrieval.
This is a replication of the methodology applied by Linkup for his linkup-sparseup-embed-v1 model except that instead of English, we're applying it to French.
The resulting model becomes the new SOTA for sparse embedding models in French (see the results section).
More details about are available in this 📝 blog post and this 🤗 HF collection.

Model Details

Model Description

Full Model Architecture

SparseEncoder(
  (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'query_length': 256, 'document_length': 1024, 'architecture': 'ModernBertForMaskedLM'})
  (1): SparseUpPooling({'logit_shift': 2.0, 'position_top_k': 12, 'vocab_fold': 'case_space', 'chunk_size': 32, 'prefix_token_ids': [[50238], [50239]], 'special_token_ids': [1, 2, 3, 4, 50238, 50239], 'embedding_dimension': 50240, 'pool_factor': 1})
)

Evaluation

Metrics

Below, the blue cells correspond to the best model across all embedding models categories, while the green cells correspond to the best sparse model for French.

Truncation configuration

Sliding window configuration

Usage

Sentence Transformers

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder
model = SparseEncoder("alphaedge-ai/sparseup-fr-v0", trust_remote_code=True)
query = [
    "Quel est le salaire moyen d'un ingénieur en France ?",
]
documents = ["En 2024, le salaire médian d'un ingénieur diplômé en France tourne autour de 45 000 euros brut annuels en début de carrière, et dépasse souvent 60 000 euros après dix ans."]
q = model.encode_query(query)
print(model.decode(q, top_k=10))

# [[('▁salaire', 2.640625), ('▁france', 2.515625), ('énieur', 2.46875), ('▁ing', 2.390625), ...]]

Diagnosis

format_bag

import torch
from sparseup import format_bag, l0

with torch.inference_mode():
    q = model.encode_query([query], convert_to_tensor=True, show_progress_bar=False)
    d = model.encode_document(documents, convert_to_tensor=True, show_progress_bar=True)

score = float(model.similarity(q, d)[0, 0])
print(f"dot(query, doc0) = {score:.3f}   L0 query = {int(l0(q)[0])}   L0 doc0 = {int(l0(d[0]))}")

print("\nquery :", QUERY)
print(format_bag(model.decode(q[0], top_k=15)))

dot(query, doc0) = 135.328   L0 query = 68   L0 doc0 = 227

query : Quel est le salaire moyen d'un ingénieur en France ?
  salaire                2.65 ██████████
  france                 2.51 ██████████
  énieur                 2.48 █████████
  ing                    2.40 █████████
  énieurs                2.26 █████████
  engineering            2.17 ████████
  average                2.12 ████████
  french                 2.10 ████████
  pay                    2.07 ████████
  França                 2.03 ████████
  salari                 2.02 ████████
  françaises             1.95 ███████
  Francia                1.91 ███████
  française              1.87 ███████
  français               1.86 ███████

highlight_html

from IPython.display import HTML, display
from sparseup import highlight_html

doc0 = DOCUMENTS[0]
display(HTML("<b>Highlight</b>" + highlight_html(doc0, model.decode(d[0]))))

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.05
• bf16: True
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: {'use_reentrant': False}
• per_device_eval_batch_size: 4
• dataloader_num_workers: 4
• prompts: {'query': '[Q] ', 'positive': '[D] ', 'negative_0': '[D] ', 'negative_1': '[D] ', 'negative_2': '[D] ', 'negative_3': '[D] ', 'negative_4': '[D] ', 'negative_5': '[D] ', 'negative_6': '[D] '}
• batch_sampler: no_duplicates

All Hyperparameters

• per_device_train_batch_size: 8
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: {'use_reentrant': False}
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 4
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: {'query': '[Q] ', 'positive': '[D] ', 'negative_0': '[D] ', 'negative_1': '[D] ', 'negative_2': '[D] ', 'negative_3': '[D] ', 'negative_4': '[D] ', 'negative_5': '[D] ', 'negative_6': '[D] '}
• batch_sampler: no_duplicates
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Citations

Sparseup-fr-v0

@misc {hf_blogpost_french_sparseup,   
    author       = { {BOURDOIS, Loïck} },  
    organization  = { {AlphaEdge AI} },  
    title        = { Sparseup-fr: A French replication of Linkup's sparse embedding model},  
    year         = 2026,  
    url          = { https://huggingface.co/blog/alphaedge-ai/french-sparseup },  
    publisher    = { Hugging Face }
}

Linkup-Platform/linkup-sparseup-embed-v1

Introducing SPARSEUP by Linkup: a move towards more competitive open sparse models by Thibault FORMAL, Pierre ERBACHER and Cosimo RULLI (2026)
https://www.linkup.so/blog/introducing-sparseup-by-linkup

mDenseOn, mLateOn, training data

@misc{sourty2026denseonlateonfullyopen,
  title         = {DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search},
  author        = {Raphaël Sourty and Antoine Chaffin and Paulo Roberto Moura Junior and Amélie Chatelain},
  year          = {2026},
  eprint        = {2607.27178},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2607.27178},
}

mmBERT

@misc{marone2025mmbertmodernmultilingualencoder,
      title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}, 
      author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2509.06888},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.06888}, 
}

opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1

@misc{geng2025competitivesearchrelevanceinferencefree,
      title={Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers}, 
      author={Zhichao Geng and Yiwen Wang and Dongyu Ru and Yang Yang},
      year={2025},
      eprint={2411.04403},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2411.04403}, 
}

sparse-encoder/splade-camembert-base-v2

splade-camembert-base-v2 by Arthur BRESNU (2025)
https://huggingface.co/sparse-encoder/splade-camembert-base-v2

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedSpladeLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
@misc{formal2022distillationhardnegativesampling,
    title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
    author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and St\'ephane Clinchant},
    year={2022},
    eprint={2205.04733},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
}

SparseMultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}