AntiBERTy
AntiBERTy is an antibody-specific transformer language model pre-trained on 558M natural antibody sequences from the Observed Antibody Space, as described in Deciphering antibody affinity maturation with language models and weakly supervised learning. It is an 8-layer BERT encoder (512 hidden units, 8 attention heads, 26M parameters) trained with masked language modelling together with species and chain-type classification heads, and provides the sequence features used by IgFold for antibody structure prediction.
Files
| File | Description |
|---|---|
model.safetensors | Model weights (fp32, 104 MB) |
config.json | transformers BertConfig for the encoder |
vocab.txt | 25-token vocabulary: [PAD] [UNK] [CLS] [SEP] [MASK] + 20 amino acids, one token per residue |
Usage
Install the antiberty package; the weights are
downloaded from this repository on first use and cached.
pip install antiberty
antiberty classify EVQLVQSGPEVKKPGTSVKVSCKASGFTFMSSAVQWVRQARGQRLEWIGWIVIGSGNTNYAQKFQERVTITRDMSTSTAYMELSSLRSEDTAVYYCAAPYCSSISCNDGFDIWGQGTMVTVS
from antiberty import AntiBERTyRunner
antiberty = AntiBERTyRunner()
embeddings = antiberty.embed(["EVQLVQSGPEVKKPGTSVKVSCKASGFTFMSSAVQWVRQARGQRLEWIGWIVIGSGNTNYAQKFQERVTITRDMSTSTAYMELSSLRSEDTAVYYCAAPYCSSISCNDGFDIWGQGTMVTVS"])
species, chains = antiberty.classify([...])
filled = antiberty.fill_masks(["QVQLQESGGGLVQAGGSLTLSCAVSG__FSNYAMG"])
The model class (antiberty.AntiBERTy, a transformers.BertPreTrainedModel with the extra
classification heads) and the tokenizer live in the package; the weights can also be loaded with
AntiBERTy.from_pretrained(<local snapshot>, attn_implementation="eager").
Intended use and limitations
AntiBERTy expects antibody variable-domain sequences (roughly 100-130 residues per chain, maximum 510). It was trained on heavy and light chains from human, mouse, rat, rabbit, rhesus and camelid repertoires; outputs for other proteins or full-length antibody chains are not meaningful.
Citation
@article{ruffolo2021deciphering,
title = {Deciphering antibody affinity maturation with language models and weakly supervised learning},
author = {Ruffolo, Jeffrey A and Gray, Jeffrey J and Sulam, Jeremias},
journal = {arXiv preprint arXiv:2112.07782},
year= {2021}
}