HELM-BERT
A peptide language model using HELM (Hierarchical Editing Language for Macromolecules) notation, compatible with Hugging Face Transformers.
Model Description
HELM-BERT is built upon the DeBERTa architecture, pre-trained on ~75k peptides from four databases (ChEMBL, CREMP, CycPeptMPDB, Propedia) using Masked Language Modeling (MLM) with a Warmup-Stable-Decay (WSD) learning rate schedule.
- Disentangled Attention: Decomposes attention into content-content and content-position terms
- Enhanced Mask Decoder (EMD): Injects absolute position embeddings at the decoder stage
- Span Masking: Contiguous token masking with geometric distribution
- nGiE: n-gram Induced Encoding layer (1D convolution, kernel size 3)
Model Specifications
| Parameter | Value |
|---|---|
| Parameters | 54.8M |
| Hidden size | 768 |
| Layers | 6 |
| Attention heads | 12 |
| Vocab size | 78 |
| Max token length | 512 |
| Pre-training data | ~75k peptides (ChEMBL, CREMP, CycPeptMPDB, Propedia) |
| Pre-training objective | MLM (span masking, p=0.15) |
| LR schedule | Warmup-Stable-Decay (WSD) |
How to Use
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("Flansma/helm-bert", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Flansma/helm-bert", trust_remote_code=True)
# Cyclosporine A
inputs = tokenizer("PEPTIDE1{[Abu].[Sar].[meL].V.[meL].A.[dA].[meL].[meL].[meV].[Me_Bmt(E)]}$PEPTIDE1,PEPTIDE1,1:R1-11:R2$$$", return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state
Training Data
Pre-trained on deduplicated peptide sequences from:
- ChEMBL: Bioactive molecules database
- CREMP: Cyclic peptide conformational ensemble database
- CycPeptMPDB: Cyclic peptide membrane permeability database
- Propedia: Protein-peptide interaction database
Downstream Performance
Permeability Regression (CycPeptMPDB)
Single-Assay (mixed PAMPA/Caco-2 target):
| Split | R² | Pearson | RMSE | MAE |
|---|---|---|---|---|
| Random | 0.658 | 0.817 | 0.471 | 0.300 |
| Scaffold | 0.502 | 0.723 | 0.450 | 0.324 |
Per-Assay (separate models for PAMPA and Caco-2):
| Split | Assay | R² | Pearson | RMSE | MAE |
|---|---|---|---|---|---|
| Random | PAMPA | 0.800 | 0.895 | 0.355 | 0.253 |
| Random | Caco-2 | 0.747 | 0.866 | 0.388 | 0.289 |
| Scaffold | PAMPA | 0.529 | 0.739 | 0.412 | 0.295 |
| Scaffold | Caco-2 | 0.637 | 0.874 | 0.405 | 0.334 |
Train/test 9:1, val 10% from train. Scaffold split by Murcko scaffolds.
PPI Classification (Propedia v2)
| Split | ROC-AUC | PR-AUC | F1 | MCC | Balanced Acc |
|---|---|---|---|---|---|
| Random | 0.968 | 0.901 | 0.847 | 0.808 | 0.906 |
| aCSM | 0.862 | 0.683 | 0.587 | 0.522 | 0.722 |
Train/test 8:2, val 10% from train, 1:4 positive:negative ratio.
- Random: random split
- aCSM: clustering-based split on aCSM-ALL complex signatures with protein overlap pruning
PPI Classification (ChEMBL)
| Split | ROC-AUC | PR-AUC | F1 | MCC | Balanced Acc |
|---|---|---|---|---|---|
| Random | 0.992 | 0.975 | 0.948 | 0.936 | 0.969 |
| Family | 0.786 | 0.449 | 0.267 | 0.222 | 0.570 |
Val 10% from train.
Citation
@article{lee2026helmbert,
title={HELM-BERT: Topology-Aware Representations for Chemically Modified Peptides},
author={Lee, Seungeon and Koyama, Takuto and Maeda, Itsuki and Matsumoto, Shigeyuki and Okuno, Yasushi},
journal={Journal of Chemical Information and Modeling},
year={2026},
doi={10.1021/acs.jcim.6c00451},
publisher={American Chemical Society},
url={https://pubs.acs.org/doi/10.1021/acs.jcim.6c00451}
}
License
MIT License