ProtSent-V2 ESM-C 300M
Contrastively fine-tuned ESM-C 300M producing fixed-length protein embeddings where biological similarity maps to embedding proximity. Intended for retrieval, clustering, and nearest-neighbour transfer.
This is the ProtSent V2 recipe applied to an ESM-C backbone.
Continued training from the Synthyra/ESMplusplus_small (ESMC-300M) weights.
Usage
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("GrimSqueaker/ProtSent-V2-ESMC-300M", trust_remote_code=True)
emb = model.encode([
"MKTLLLTLVVVTIVCLDLGYT",
"MKTLLLTLVVVTIVCLDLGYN",
"AGWYRSPQEGLKPVDTFKDIV",
])
print(cos_sim(emb[0], emb[1:]))
Embeddings are mean-pooled over the final layer, dimension 960. No Matryoshka heads on this release (see Training configuration).
Training data
Four sources. Pfam/AFDB/STRING are decontaminated against benchmark test sets (see below); the DMS source is not — see Limitations.
| source | pairs used |
|---|---|
| Pfam families | 1,230,551 |
| AlphaFold DB (Foldseek clusters) | 27,105,179 |
| STRING-DB v12 PPI | 15,000,000 |
| ProteinGym DMS (CoSENT target) | 1,000,000 |
| total | 44,335,730 |
Pfam/AFDB pairs are drawn with the disjoint-pair sampler at up to 10 pairs per cluster
(k=10), which spends the pair budget on distinct sequence pairs rather than repeating
members — covers up to 90 distinct sequences per cluster instead of 10. STRING is a fixed
15M-pair subsample (seed 42).
Decontamination
Pfam, AlphaFold DB and STRING were searched against the benchmark test sequences with
MMseqs2 easy-search (40% identity, 80% coverage, --cov-mode 1) and matching sequences
removed before training — the same protsent-data-dc40 corpus used by the ProtSent-V2.5
ESM-2 releases.
Training configuration
| setting | value |
|---|---|
| backbone | ESM-C 300M (Synthyra/ESMplusplus_small, 960 hidden) |
| loss | CachedMultipleNegativesRankingLoss (Pfam/AFDB/STRING) + CoSENT (DMS) |
| contrastive batch | 2048 per device |
| gather across devices | off |
| multi-dataset sampler | proportional, world-size-aligned (see below) |
| GOR weight | 0 (off) |
| Matryoshka | off |
| max sequence length | 512 |
| optimiser | AdamW, LR 1e-4, warmup 300 steps |
| precision / attention | bf16, flash-attention-2 (FA3 unavailable on this hardware) |
| hardware | 3x NVIDIA B300 |
| steps | 7,216 (one epoch) |
| gradient-cache mini-batch | 128 |
Training code: github.com/oriel9p/ProtSent,
train_esmc_300m_v2.sh.
Results
23-task suite, -e test, seed 42, both kNN and linear-probe transfer. Full per-task CSVs:
results/benchmarks/ism/protsent_esmc_300m_v2_{knn,linear}/ in the repo above.
SCOPe-40 structural retrieval, eligible queries only
| method | R@1 | R@10 | MAP |
|---|---|---|---|
| ESM-C 300M (this model's backbone, untrained) | 0.3709 | 0.5794 | 0.2212 |
| ISM-C 300M (structure-distilled ESM-C 300M) | 0.4300 | 0.6592 | 0.2733 |
| HMMER (phmmer, filters off) | 0.7525 | 0.8978 | 0.6067 |
| ProtSent-V2 ESM-C 300M | 0.7974 | 0.9539 | 0.7692 |
23-task suite vs the model's own untrained backbone and vs ISM-C
Win/tie/loss, tie tolerance 0.005:
| probe | vs ESM-C 300M (this backbone, untrained) | vs ISM-C 300M |
|---|---|---|
| kNN | 16W / 0T / 7L | 14W / 1T / 8L |
| linear | 6W / 3T / 14L | 6W / 4T / 13L |