bert-tiny Semantic Similarity / Paraphrase Detection
A fine-tuned prajjwal1/bert-tiny (4.4M parameters, 2 layers, hidden size 128) cross-encoder for sentence-pair semantic similarity / paraphrase / duplicate-question detection. Trained jointly on STS-B, QQP, PAWS, and MRPC.
Model Details
- Base model: prajjwal1/bert-tiny (L=2, H=128, ~4.4M params)
- Architecture: Cross-encoder,
BertForSequenceClassification(2-class) - Task: Binary classification — given two sentences, predict whether they are semantically similar / duplicate / paraphrase (label 1) or not (label 0)
- Max sequence length: 64 tokens (sentence1 + sentence2 combined)
- License: MIT (matches base model)
Training Data
Trained on the concatenation of four datasets (label conversion applied where needed):
| Dataset | Size (train) | Label rule |
|---|---|---|
| STS-B (GLUE) | 5,749 | Continuous score ≥ 3.0 → 1, else 0 |
| QQP (GLUE) | 363,846 | Already binary (duplicate question) |
| PAWS (labeled_final) | 49,401 | Already binary (paraphrase) |
| MRPC (GLUE) | 3,668 | Already binary (paraphrase) |
License note: QQP's own terms are primarily intended for research/non-commercial use. If you plan commercial deployment, please review QQP's original license independently — the fine-tuned weights here are MIT-licensed, but the training-data provenance is documented for transparency.
Training Configuration
- Optimizer: AdamW, learning rate 1e-4
- Batch size: 32
- Epochs: 10
- Loss: Class-weighted CrossEntropyLoss (weights inverse-proportional to label frequency)
- Label smoothing: 0.1
- Warmup ratio: 0.1, weight decay: 0.01
Usage
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/YOUR_MODEL_NAME")
model = AutoModelForSequenceClassification.from_pretrained("YOUR_USERNAME/YOUR_MODEL_NAME")
model.eval()
def similarity(sent1, sent2):
inputs = tokenizer(sent1, sent2, return_tensors="pt", truncation=True, max_length=64, padding="max_length")
with torch.no_grad():
logits = model(**inputs).logits
prob = torch.softmax(logits, dim=-1)[0][1].item()
return prob # probability that the two sentences are similar/duplicate
print(similarity("How old are you?", "What is your age?"))
# -> high probability (similar)
Evaluation Results
Evaluated on the official validation/test splits of each dataset (threshold = 0.5):
| Dataset | Accuracy | F1 | Spearman |
|---|---|---|---|
| QQP (validation) | 81.4% | 77.5% | — |
| MRPC (validation) | ~70–72% | ~80–82% | — |
| STS-B (validation) | — | — | ~0.665 |
| PAWS (test) | ~51% | ~55% | — |
Known Limitation — PAWS / Adversarial Word-Order Sensitivity
This model, like most compact (sub-15M parameter) transformer encoders, performs close to chance level (~50%) on PAWS, which specifically tests whether a model can distinguish sentence pairs with high lexical overlap but different meaning due to word order (e.g., "The cat chased the mouse" vs. "The mouse chased the cat").
Extensive experimentation (data rebalancing, focal loss, hard-negative augmentation, knowledge distillation from larger sentence-embedding teachers, and a separate bi-encoder + contrastive-loss architecture with anchor-matched hard-negative mining) was unable to move PAWS performance meaningfully above chance level. This appears to be a capacity-bound limitation of shallow (2-layer, small hidden-size) transformer encoders rather than a data or training-strategy issue — larger models (BERT-base and above) trained on PAWS reach 85–94% on the same benchmark.
Practical implication: this model is reliable for general semantic similarity / duplicate-question / paraphrase detection (QQP, MRPC, STS-B-style tasks), but should not be relied upon for adversarial or word-order-sensitive discrimination tasks.
Intended Use
- Lightweight semantic similarity scoring where model size/latency matters more than adversarial robustness
- Duplicate question detection, FAQ matching, basic paraphrase detection
- Not recommended for: legal/compliance-grade paraphrase detection, or any use case where word-order-reversal attacks are a realistic concern