PlantPLM-150M
ESM-2 150M parameter model continued-pretrained on Viridiplantae (plant) protein sequences.
This is a domain-adapted version of facebook/esm2_t30_150M_UR50D, fine-tuned on a non-redundant subset of UniProt TrEMBL plant-kingdom proteins.
Part of the Plant-PLM - ESM-2 models at 8M, 35M, 150M, and 650M parameters, each adapted on plant protein data.
Model Description
| Property | Value |
|---|---|
| Base model | facebook/esm2_t30_150M_UR50D |
| Architecture | ESM-2 · 30 layers · hidden=640 · heads=20 · FFN=2560 |
| Position embeddings | Rotary (RoPE) |
| Vocabulary | 33 tokens (20 standard + rare amino acids + special tokens) |
| Parameters | 148M (full-parameter continued pretraining) |
| Training objective | Masked Language Modeling (MLM, 15% masking) |
Training Data
Unlike the 8M and 35M variants (trained on the raw, redundant plant TrEMBL corpus), this model was trained on a redundancy-reduced ("nr50") corpus: the raw Viridiplantae corpus was clustered with MMseqs2 easy-linclust (50% identity / 80% coverage, mirroring ESM-2's own training-data construction) and one representative sequence per cluster was kept.
| Property | Value |
|---|---|
| Source | UniProt TrEMBL — Viridiplantae (plant kingdom) subset, MMseqs2-deduplicated (50% ID / 80% cov) |
| Sequences | 4,372,758 (down from 19,938,415 raw, −78%) |
| Avg sequence length | 279 AA · median 199 AA |
| Token budget | ~1.11 billion amino acid tokens (≈ 1 full epoch over the nr50 corpus) |
Training Details
| Hyperparameter | Value |
|---|---|
| Training steps | 90,000 optimizer steps (1 epoch over nr50) |
| Batch size | 48 sequences (12 per micro-batch × 4 gradient accumulation steps) |
| Optimizer | AdamW · β=(0.9, 0.98) · ε=1e-8 · weight_decay=0.01 |
| Learning rate | 1e-5 |
| LR schedule | Linear warmup (1,000 steps) → linear decay |
| Gradient clipping | 1.0 |
| Precision | 16-bit mixed |
| Gradient checkpointing | Enabled |
| Hardware | 1× NVIDIA RTX 3060 (12 GB) |
Final metrics (validation set, 5% holdout):
| Metric | Value |
|---|---|
val/mlm_loss | 2.185 |
val/perplexity | 8.98 |
val/masked_token_acc | 34.3% |
Usage
from transformers import EsmForMaskedLM, EsmTokenizer
import torch
model = EsmForMaskedLM.from_pretrained("dipayan26/PlantPLM-150M")
tokenizer = EsmTokenizer.from_pretrained("dipayan26/PlantPLM-150M")
# --- Masked token prediction ---
sequence = "MSPQTETKASVGFKAGVKDYKLTYYTPEYETK"
inputs = tokenizer(sequence, return_tensors="pt")
# mask one position
inputs["input_ids"][0, 5] = tokenizer.mask_token_id
with torch.no_grad():
logits = model(**inputs).logits
masked_pos = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero()[0, 1]
top5 = logits[0, masked_pos].topk(5)
print(tokenizer.convert_ids_to_tokens(top5.indices.tolist()))
# --- Sequence embedding ([CLS] token) ---
inputs = tokenizer(sequence, return_tensors="pt")
with torch.no_grad():
hidden = model.esm(**inputs).last_hidden_state
cls_embedding = hidden[0, 0, :] # shape: [640]
print("Embedding shape:", cls_embedding.shape)
Intended Use
- Plant protein function prediction — GO term annotation, subcellular localization, signal peptide detection
- Plant-specific protein embeddings — clustering, retrieval, similarity search
- Transfer learning starting point — fine-tune on small labeled plant protein datasets
Out-of-scope Use
- Non-plant organisms — the model has been shifted toward Viridiplantae statistics; use the original
facebook/esm2_t30_150M_UR50Dfor general protein tasks - Structural prediction — not trained for structure; use ESMFold for that
Limitations
- No downstream benchmark evaluation has been run on this checkpoint yet
Citation
If you use this model, please cite:
@misc{sarkar2026plantplm,
author = {Sarkar, Dipayan},
title = {PlantPLM: Domain-Adaptive Pretraining of ESM-2 on Viridiplantae Proteins},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/dipayan26/PlantPLM-150M}},
}