minbert-sts-nli-meanpool
A bert-base-uncased encoder fine-tuned in two stages for Semantic Textual
Similarity (STS-B style regression, labels in [0, 5]):
- NLI pretraining. The bi-encoder is first trained as a Siamese sentence encoder on the combined SNLI + MultiNLI entailment task (942,069 train pairs), giving it a head start at telling semantically related sentences apart before ever seeing an STS label.
- STS fine-tuning. The NLI-pretrained encoder is then fine-tuned on the STS training set: all layers unfrozen, lr 5e-6, batch size 32, 100-step linear warmup, weight decay 0.05, dropout 0.2, up to 4 epochs with early-stopping patience 2.
Scoring. There is no learned similarity head. A sentence is represented by
masked mean pooling over last_hidden_state (including the [CLS]/[SEP]
positions), and the similarity score is the cosine of the two sentence
vectors, rescaled from [-1, 1] to [0, 5]:
score = (cosine_similarity(mean_pool(s1), mean_pool(s2)) + 1) * 2.5
Mean pooling was chosen over the [CLS]/pooler-output representation because it outperformed CLS at every layer in a pooling-strategy sweep on this project's STS dev set.
Results (STS-B dev set)
| Seed | Pearson | Spearman |
|---|---|---|
| 11711 | 0.7432 | 0.7457 |
| 2026 | 0.7436 | 0.7460 |
| 42 | 0.7388 | 0.7411 |
| mean (n=3) | 0.7419 | 0.7443 |
The seed=11711 checkpoint published here (Pearson 0.7432, Spearman 0.7457) is the one submitted for this project; report the 3-seed mean above, not the single-seed number, as the figure the method should be judged on -- std over 3 seeds is a rough sense of spread, not a precise estimate.
Usage
This checkpoint stores only the BERT encoder (bert.*); there is no separate
classification head to load. To reproduce the pooling/scoring exactly, use
this project's bert.BertModel and multitask_classifier.masked_mean_pool
(https://github.com/nirish2407/dnlp), or replicate the same masked mean
pooling + cosine recipe with any BERT loader.
Training details
See config.json in this repository for the full recorded training recipe
(learning rate, epochs, dropout, warmup, weight decay, and the NLI checkpoint
this model was initialised from), and the STS section of the project README
(https://github.com/nirish2407/dnlp) for the full experimental writeup,
including the ablations that motivated each choice (NLI-pretraining control
run, pooling-strategy sweep, layer-wise readout analysis, and a residualized
probe checking the gains are not just lexical overlap).