SPARSEUP
TL;DR: SPARSEUP is a strong and efficient sparse retriever trained on the same backbone family and fine-tuning data as DenseOn / LateOn (149M, ModernBERT). Strongest public vocabulary-based sparse encoder we know of under 150M on BEIR-13 (56.4 nDCG@10). Apache 2.0. Use it!
For the full story, see the blog post.
About
LightOn released open data and a recipe to train DenseOn (single-vector) and LateOn (late interaction). SPARSEUP fills the missing slot: a learned sparse embedding model, so the three architectures can be compared with backbone and data held fixed. The model is initialized from LateOn-unsupervised with ModernBERT’s MLM head / decoder.bias grafted on (decoder.weight stays tied to LateOn embeddings). Contrastive fine-tuning on LightOn’s mixture (7 hard negatives sampled from 50, plus in-batch). No cross-encoder distillation.
Three extra knobs vs vanilla SPLADE-like model:
logit_shift=15—log(1 + ReLU(x − 15))so ReLU support starts sparseposition_top_k=12— each token keeps 12 vocab dims max before pooling (expansion budget)vocab_fold=case_space— fold case /Ġsurface forms onto one id after pooling (avoid duplication of surface forms = improve efficiency). Voc goes from 50k to ~34k.
Prefixes [Q] / [D] are attended but not pooled. Scoring is dot product.
BEIR (nDCG@10)
-
Controlled comparison against LateOn and DenseOn (same backbone / data). SPARSEUP uses Seismic (approximate); their numbers are exact search. | Model | Avg | Avg (w/o MS MARCO) |
| --- | --- | --- |
| LateOn | 57.9 | 58.9 |
| DenseOn | 56.9 | 57.9 |
| SPARSEUP | 55.4 | 56.4 | -
Comparison against competitive sparse encoders. Best model at <150M parameters. | Model | Avg (wo/ MS MARCO) | | --- | --- | | splade-v3 | 51.7 | | granite-embedding-30m-sparse | 50.6 | | ModernBERT-VT | 52.4 | | opensearch-neural-sparse-encoding-v1 | 52.4 | | opensearch-neural-sparse-encoding-doc-v3-gte | 54.6 | | SPARSEUP | 56.4 | | LACONIC-1B** | 58.7 |
SPARSEUP is fast
Using dedicated infra (Seismic), it can reach sub-ms latency on MS MARCO (single-threaded).
Model description
- Model Type: Sentence Transformer or bare Transformers (see below)
- Encoder Type (sparse): MLM →
log1p(ReLU(x − θ))→ per-position top-k → max-pool → vocab fold - Output Dimensionality: ~34k (sparse)
- Base: ModernBERT-base (149M), from LateOn-unsupervised + grafted MLM head
- Language: English
- Similarity: dot
- Query / doc prefixes:
[Q]/[D] - Max lengths (eval): query 128, document 512
- License: Apache 2.0
Usage
Transformers
from transformers import AutoModel
model = AutoModel.from_pretrained("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
Encoding
encode_to_dict(texts, kind="document", top_k=None) — encodes texts and returns one {token: weight} dict per text, sorted by descending weight. kind ("query"/"document") picks the prefix and max length; top_k caps how many terms you get back (None = all non-zero).
queries = ["England football player highest paid", "NYC capital which country?"]
model.encode_to_dict(queries, kind="query", round_to=2, top_k=10)
[{'ĠEngland': 2.96,
'Ġfootball': 2.68,
'Ġpaid': 2.65,
'Ġplayer': 2.57,
'Ġhighest': 2.52,
'Ġplayers': 2.19,
'Ġpay': 2.04,
'Ġenglish': 2.03,
'Ġeng': 2.03,
'Ġfootballer': 1.97},
{'Ġcapital': 3.25,
'ĠNYC': 3.12,
'Ġcap': 3.12,
'Ġcountry': 3.12,
'Ġny': 3.09,
'ĠYork': 2.87,
'Ġcountries': 2.59,
'Ġcapit': 2.55,
'Ġnation': 2.47,
'ĠCapitol': 2.45}]
Scoring
documents = [
"Harry Kane is the England captain and among the highest-paid footballers.",
"New York City is in the United States, not a national capital.",
]
q = model.encode(queries, kind="query")
d = model.encode(documents, kind="document")
print(model.score(q, d))
#tensor([[113.2993, 0.0000],
# [ 7.0030, 99.1290]])
Diagnosis
We also provide utilities to better visually inspect the content of predictions.
- prediction with attributions:
render(texts, kind="document", top_k=25, width=36, color=None)— returns a terminal bar chart string: one block per text, one line per expansion term with a heat-colored bar proportional to weight, plus the input subtoken that produced it (<- source@pos, for pure expansions).widthis the bar length in characters;color=Noneauto-detects a TTY, so passcolor=Truein Jupyter.
print(model.render(queries, kind="query", top_k=10))
query · England football player highest paid
England █████████████████████████████████▓▒░ 2.96 <- ·England@2
football ██████████████████████████████▓▒░ 2.68 <- football@3
paid █████████████████████████████▓▒░ 2.65 <- paid@6
player ████████████████████████████▓▒░ 2.57 <- player@4
highest ████████████████████████████▓▒░ 2.52 <- highest@5
players ████████████████████████▓▒░ 2.19 <- player@4 <exp>
pay ██████████████████████▓▒░ 2.04 <- paid@6 <exp>
english ██████████████████████▓▒░ 2.03 <- ·England@2 <exp>
eng ██████████████████████▓▒░ 2.03 <- ·England@2 <exp>
footballer █████████████████████▓▒░ 1.97 <- player@4 <exp>
query · NYC capital which country?
capital █████████████████████████████████▓▒░ 3.25 <- capital@4
NYC ████████████████████████████████▓▒░ 3.12 <- ·C@3 <exp>
cap ████████████████████████████████▓▒░ 3.12 <- capital@4 <exp>
country ████████████████████████████████▓▒░ 3.12 <- country@6
ny ███████████████████████████████▓▒░ 3.09 <- ·NY@2
York █████████████████████████████▓▒░ 2.87 <- ·NY@2 <exp>
countries ██████████████████████████▓▒░ 2.59 <- country@6 <exp>
capit █████████████████████████▓▒░ 2.55 <- capital@4 <exp>
nation ████████████████████████▓▒░ 2.47 <- country@6 <exp>
Capitol ████████████████████████▓▒░ 2.45 <- capital@4 <exp>
- visual attribution:
highlight(texts, kind="document", reduce="sum", color=None)— returns the original texts with their firing words lit up, intensity being each word's share of the vector's total mass (reduce="sum", default) or its single strongest term (reduce="max"). On a TTY it uses reverse-video heat colors; in plain mode it wraps words in tiered markers ⟦strong⟧ «mid» ‹weak› (samecolor=Truecaveat for Jupyter).
document = ["The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent lives obliterated."]
print(model.highlight(document, kind="document", color=True))
# type: use color=True for a more visual rendering. ⟦ ⟧ / « » / ‹ › represent strong / mid / weak attributions.
‹The› ⟦presence⟧ «of» ⟦communication⟧ ⟦amid⟧ ⟦scientific⟧ ⟦minds⟧ «was» ⟦equally⟧ ⟦important⟧ «to» the ⟦success⟧ «of» «the» ⟦Manhattan⟧ ⟦Project⟧ «as» ‹scientific› ⟦intellect⟧ was«.» ‹The› «only» ⟦cloud⟧ ⟦hanging⟧ «over» the ⟦impressive⟧ ⟦achievement⟧ of the ⟦atomic⟧ ⟦researchers⟧ ‹and› ⟦engineers⟧ ‹is› what their «success» «truly» ⟦meant;⟧ ⟦hundreds⟧ «of» ⟦thousands⟧ «of» ⟦innocent⟧ ⟦lives⟧ ⟦obliterated⟧.
Sentence Transformers
The model can be used with Sentence Transformers. First install the lib:
pip install -U sentence-transformers
from sentence_transformers import SparseEncoder
model = SparseEncoder("Linkup-Platform/linkup-sparseup-embed-v1", trust_remote_code=True)
queries = [
"England football player highest paid",
"NYC capital which country?",
]
q = model.encode_query(queries)
print(model.decode(q, top_k=10))
# [[('ĠEngland', 2.96), ('Ġfootball', 2.68), ...],
# [('Ġcapital', 3.25), ('ĠNYC', 3.12), ...]]
documents = [
"Harry Kane is the England captain and among the highest-paid footballers.",
"New York City is in the United States, not a national capital.",
]
d = model.encode_document(documents)
print(model.similarity(q, d))
# tensor([[113.29, 0.00],
# [ 7.00, 99.13]])