🧬 Biomedical NER — Experimental Model (BC5CDR)

This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract on the bc5cdr dataset. It achieves the following results on the evaluation set:

• Loss: 0.0835
• Precision: 0.8582
• Recall: 0.8977
• F1: 0.8775
• Accuracy: 0.9727

This repository hosts a testing version of a biomedical named entity recognition (NER) model fine-tuned on the BC5CDR dataset, which contains PubMed abstracts annotated for diseases and chemicals.

⚠️ Note: This model is shared for educational and testing purposes only. It is not intended for clinical or production use and may not generalize beyond the BC5CDR dataset.

Model overview

• Base model: PubMedBERT (abstracts) (domain-specific transformer)
• Task: Named Entity Recognition (token classification)
• Entities:
  • Chemical
  • Disease
  • O (non-entity tokens)

The model predicts entity spans following the standard BIO tagging scheme, e.g. B-Chemical, I-Disease, O.

Training notebook

The full training and preprocessing pipeline — including dataset preparation, sentence splitting, BIO tagging, and evaluation — is documented in this Kaggle notebook

Example usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("Francesco-A/BiomedNLP-PubMedBERT-base-uncased-abstract-bc5cdr-ner-v1")
model = AutoModelForTokenClassification.from_pretrained("Francesco-A/BiomedNLP-PubMedBERT-base-uncased-abstract-bc5cdr-ner-v1")

ner = pipeline("token-classification", model=model, tokenizer=tokenizer, aggregation_strategy="first")

demo_texts = [
    "Aspirin is often used to treat inflammation, but may cause gastric bleeding.",
    "Naloxone reverses the antihypertensive effect of clonidine.",
]

for t in demo_texts:
    print("\nText:", t)
    preds = ner(t)
    for p in preds:
        print(f'  - {p["word"]}: {p["entity_group"]} (score {p["score"]:.2f})')

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-5
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 8
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.45.0
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.20.3