pubmedbert-pmid-tapt
Domain-adapted PubMedBERT (a.k.a. BiomedBERT) for the kinase / small-molecule drug-discovery literature. The base encoder was continue-pretrained with masked language modelling (MLM) — i.e. task-adaptive pretraining (TAPT) — on unlabelled, ChEMBL-linked PubMed abstracts. It is intended as an encoder backbone for downstream biomedical information extraction (NER + relation extraction of potency/binding claims), not as a standalone predictor.
- Base model:
microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext - Adaptation: self-supervised MLM (15% dynamic masking), no labels
- Domain: kinase inhibitors, IC50/Ki/Kd/EC50 potency, medicinal-chemistry abstracts
- Architecture: BERT encoder (
BertForMaskedLM), unchanged from the base
Intended use & limitations
Intended use. Fine-tune or feature-extract for biomedical extraction tasks — chemical/gene
NER, (compound, relation, target) relation extraction, and atomic-fact-claim (AFC) extraction
in a fact-verification pipeline. Usable directly for fill-mask probing.
Out of scope. Not a clinical, diagnostic, or safety tool. It does not verify facts by itself and must not be used for medical decisions. Numeric spans (values/units) are subword- tokenised, so the encoder is weak at exact numeric reasoning — bind measurements with rules downstream.
Training data
- Corpus:
pmid_coverage_corpus.jsonl— PubMed title + abstract text for PMIDs linked to curated ChEMBL kinase activities (produced by NB07 of the project). - Size: ~4,026 abstracts, packed into 1662 blocks of
512tokens; split train / val / test. - Labels: none — the text is its own supervision (self-supervised MLM).
Training procedure
| Hyperparameter | Value |
|---|---|
| Objective | Masked LM (MLM), 15% dynamic masking |
| Block size | 512 tokens |
| Epochs | 3 |
| Learning rate | 2e-05 |
| LR schedule | linear + warmup (ratio 0.06) |
| Weight decay | 0.01 |
| Optimizer | AdamW |
| Mixed precision | fp16 (GPU) |
| Seed | 42 |
| Selection metric | validation eval_loss (best checkpoint kept) |
Evaluation
Perplexity on the held-out test split (lower is better). The meaningful signal is the drop from base → adapted on the same split.
| Model | Val loss | Val ppl | Test loss | Test ppl |
|---|---|---|---|---|
| Base PubMedBERT | 1.1065 | 3.02 | 1.0444 | 2.84 |
| This model (adapted) | 0.9488 | 2.58 | 0.9332 | 2.54 |
How to use
from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
repo = "Sarbanidg/pubmedbert-pmid-tapt"
tok = AutoTokenizer.from_pretrained(repo)
mdl = AutoModelForMaskedLM.from_pretrained(repo)
fill = pipeline("fill-mask", model=mdl, tokenizer=tok)
fill("The compound inhibited EGFR with an IC50 of 12 [MASK].")
Provenance & reproducibility
Produced by notebook src/notebooks/08_pubmedbert_mlm_pretrain.ipynb (§1–§6) in the
med_llm_factcheck project. CLI/local equivalent: util/pretrain_pubmedbert_mlm.py.
License
MIT, inherited from the base model microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext.