echo-sbert-domain
A sentence embedding model for short, messy, real-world app-store reviews.
Built for Echo, which turns 100,000 Swiggy Google Play reviews into tracked themes. Trained in two stages:
- Sentence-BERT reproduction.
distilroberta-baseon 300,000 SNLI + MultiNLI pairs, siamese, mean pooling, classifier on(u, v, |u-v|)— written by hand in raw PyTorch, reproducing Reimers & Gurevych (2019). - Domain adaptation. Continued with
MultipleNegativesRankingLosson 53,061 pairs mined from the reviews themselves, where TF-IDF and the stage-1 encoder independently agree, plus SimCSE dropout self-pairs.
Results
| benchmark | score |
|---|---|
| STS average (7 datasets) after stage 1 | 72.17 |
| STS average after stage 2 | 74.54 |
| Review retrieval, Precision@10 | 61.15 |
| Review retrieval, + cross-encoder rerank | 75.77 |
| Theme assignment, blind hand-audit | 82.4% |
Stage 2 improved generic STS by +2.37 while adapting to the domain, which was predicted to degrade and did not. Note that 74.54 is not "beating the paper's 74.21": that number comes from NLI training alone, this adds a second stage.
Limitations, stated plainly
- It does not bridge Hinglish. "khana thanda tha" against "the food was cold" scores 0.066, versus 0.049 for a genuinely unrelated pair. Romanised Hindi reviews cluster by language rather than by subject.
- On its own it loses to TF-IDF for review retrieval (61.15 vs 65.00). It only wins in front of a cross-encoder reranker.
- Part of the retrieval gain may be circular — mined pairs required TF-IDF to agree, so the model may have partly learned to imitate it.
- Retrieval numbers rest on 26 hand-judged queries, one judge.
Use
from transformers import AutoModel, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("aynaval2003/echo-sbert-domain")
model = AutoModel.from_pretrained("aynaval2003/echo-sbert-domain").eval()
def embed(sentences):
x = tok(sentences, padding=True, truncation=True, max_length=128,
return_tensors="pt")
with torch.no_grad():
h = model(**x).last_hidden_state
mask = x["attention_mask"].unsqueeze(-1).float() # mean pooling,
v = (h * mask).sum(1) / mask.sum(1).clamp(min=1e-9) # ignoring padding
return torch.nn.functional.normalize(v, dim=1)
Mean pooling, and it matters — this model was trained with it. CLS pooling scores 5.1 points lower in the ablation.