Embeddings for Preferences — ST5-XL
Companion model to "Embeddings for Preferences, Not Semantics".
A sentence embedding whose distances reflect preferential similarity — the probability that a person who agrees with one piece of text agrees with nearby texts — rather than semantic similarity.
This model is a LoRA-adapted sentence-T5-XL (r=16, α=48, q/v
projections), fine-tuned with Bradley–Terry loss on counterfactual hard
triplets. The LoRA adapter has been merged into the base weights so the
model can be loaded as a regular SentenceTransformer.
Why preference instead of semantic similarity?
Two statements with nearly identical wording can take opposite stances on a political or social issue, and a participant who agrees with one will typically disagree with the other. Off-the-shelf embedding models conflate the two — surface similarity dominates the cosine score. This model is fine-tuned to suppress that nuisance signal so cosine similarity tracks who would endorse the same text.
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("cartgr/embeddings-for-preferences-st5-xl")
texts = [
"It should be legal. Religion has no place in politics if we truly want to be free.",
"It should be illegal. Religion has a place in politics if we truly want to be free.",
"To ensure true freedom, it's crucial that legislation remains secular, without religious influence.",
]
embs = model.encode(texts, normalize_embeddings=True)
import numpy as np
print("anchor vs distractor (opposite stance, same wording):", embs[0] @ embs[1])
print("anchor vs match (same stance, different wording):", embs[0] @ embs[2])
The base ST5-XL would rank the distractor closer than the match (0.87 vs
0.82); this model reverses that (0.79 vs 0.81).
Evaluation
Cosine triplet accuracy across 11 online-deliberation datasets (3 GSC, 3 Remesh, 5 Polis), mean over 5 training seeds:
| Setting | This model | Base ST5-XL | OpenAI text-emb-3-large |
|---|---|---|---|
| Natural triplets (mean of 11 datasets) | 68.6% | 65.2% | 62.9% |
| Hard counterfactual triplets (n=875) | 80.0% | 48.3% | 3.8% |
Per-dataset numbers, baselines (25 embedding models), and the full methodology are in the paper.
Training
- Base encoder:
sentence-transformers/sentence-t5-xl - Adapter: LoRA,
r=16,α=48, dropout 0.1, target modulesq, v - Loss: Bradley–Terry over cosine differences
log(1 + exp(-(cos(a, p) - cos(a, n)))) - Hyperparameters (val-selected):
lr=1.25e-4,n_hard=750, batch size 16, 46 gradient steps - Training data: 750 synthetic counterfactual hard triplets, generated with GPT-4o from a pool of 2,000 political/social issues sourced from the Habermas Machine corpus and Kialo. The training data was generated independently of the 11 evaluation datasets.
Intended uses and limitations
Intended for policy-style and opinion topics — political and social issues where stances are rooted in values (abortion, immigration, electoral reform, UBI, campus speech, etc.). Generalization to factual/scientific questions, personal-taste preferences, or abstract philosophy is untested. Best on substantive opinion-expressing texts (a sentence or longer); very short comments may not carry enough signal.
Not a stance-detection classifier, not calibrated for absolute agreement probability, and not a substitute for any individual's considered judgment on a binding decision.
Citation
To be added when the corresponding paper is publicly released.
License
Apache-2.0, inherited from the base sentence-T5-XL model.