ProtSent-V2.5 ESM-2 150M

Continued contrastive training of ProtSent-V2 150M on a fresh draw of the corpus, adding a DMS/ProteinGym CoSENT target and a Global Orthogonal Regularization term.

Mean-pooled ESM-2 150M embeddings, dimension 640, max sequence length 512.

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("GrimSqueaker/ProtSent-V2.5-150M")
emb = model.encode(["MKTLLLTLVVVTIVCLDLGYT", "MKTLLLTLVVVTIVCLDLGYN", "AGWYRSPQEGLKPVDTFKDIV"])
print(cos_sim(emb[0], emb[1:]))

Training data

Trained on 14.7M pairs (3,600 steps x 4,096 effective batch), 41% of the pool.

Pfam, AFDB and STRING are decontaminated with MMseqs2 at 40% identity / 80% coverage against the remote-homology and PPI test splits. The DMS parquet is not MMseqs2-filtered; it has zero exact-sequence overlap with the AAV Fitness (0/50,430), Stability, Variant Effect and Fluorescence test sets.

Run parameters

Training code: github.com/oriel9p/ProtSent, train_esm2_150m_v2p5.sh.

Results

SCOPe-40 structural retrieval, test split, self excluded, restricted to the 1,693 of 2,207 queries with a non-self same-family protein in the gallery.

Paired bootstrap over queries, 10,000 resamples, V2.5 − V2:

R@10, R@30 and MAP exclude zero; R@1 does not. HMMER still leads at R@1.

23-task downstream suite

Win/tie/loss over the 20 tasks with a defined one-vs-rest AUC, ties at |delta| < 0.005, with the median delta. All arms scored with --eval_split test, EvalMode=standard, seed 42.

V2.5 improves on the untuned backbone under a k-NN probe by more than V2 did. Under a linear probe, contrastive post-training costs about 0.015 against the backbone at this scale, and V2.5 does not change that.

Per-task, both V2.5 − V2 medians are inside the tie band. The movement is concentrated in the Spearman regression tasks and reverses by probe: mean delta +0.020 under k-NN and −0.006 under linear, while AUC tasks are flat under both (+0.000 and −0.001).

AAV Fitness and Stability decline monotonically across the whole 150M lineage under a linear probe (AAV 0.589 → 0.398 → 0.451 → 0.365 for ESM-2 → V1 → V2 → V2.5; Stability 0.706 → 0.699 → 0.663 → 0.639).

The AAV drop is not embedding collapse. Over 3,000 AAV test variants, V2.5 has a lower mean pairwise cosine than V2 (0.983 vs 0.992), higher effective dimensionality (10.4 vs 6.8) and a higher best-single-direction correlation with fitness (0.576 vs 0.542, against 0.554 for vanilla ESM-2). The variants are spread further apart and carry more linear signal in-sample; what degrades is transfer from the train split to the deliberately hard FLIP test split.

V2.5 changes six settings at once relative to V2 (GOR, DMS target, symmetric directions, k, seeds, batch geometry) and there is no GOR-off ablation at this scale, so no individual change is isolated here. The equivalent ablation at 35M found GOR contributed nothing measurable.