SpanMarker with bert-base-uncased on BioNLP2004
This is a SpanMarker model trained on the BioNLP2004 dataset that can be used for Named Entity Recognition. This SpanMarker model uses bert-base-uncased as the underlying encoder. See train.py for the training script.
Model Details
Model Description
- Model Type: SpanMarker
- Encoder: bert-base-uncased
- Maximum Sequence Length: 256 tokens
- Maximum Entity Length: 8 words
- Training Dataset: BioNLP2004
- Language: en
- License: other
Model Sources
Model Labels
| Label | Examples |
|---|
| DNA | "immunoglobulin heavy-chain enhancer", "enhancer", "immunoglobulin heavy-chain ( IgH ) enhancer" |
| RNA | "GATA-1 mRNA", "c-myb mRNA", "antisense myb RNA" |
| cell_line | "monocytic U937 cells", "TNF-treated HUVECs", "HUVECs" |
| cell_type | "B cells", "non-B cells", "human red blood cells" |
| protein | "ICAM-1", "VCAM-1", "NADPH oxidase" |
Evaluation
Metrics
| Label | Precision | Recall | F1 |
|---|
| all | 0.7290 | 0.7983 | 0.7621 |
| DNA | 0.7174 | 0.7505 | 0.7336 |
| RNA | 0.6977 | 0.7692 | 0.7317 |
| cell_line | 0.5831 | 0.7020 | 0.6370 |
| cell_type | 0.8222 | 0.7381 | 0.7779 |
| protein | 0.7196 | 0.8407 | 0.7755 |
Uses
Direct Use
from span_marker import SpanMarkerModel
# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-bert-base-uncased-bionlp")
# Run inference
entities = model.predict("In erythroid cells most of the transcription activity was contained in a 150 bp promoter fragment with binding sites for transcription factors AP2, Sp1 and the erythroid-specific GATA-1.")
Downstream Use
You can finetune this model on your own dataset.
from span_marker import SpanMarkerModel, Trainer
# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("tomaarsen/span-marker-bert-base-uncased-bionlp")
# Specify a Dataset with "tokens" and "ner_tag" columns
dataset = load_dataset("conll2003") # For example CoNLL2003
# Initialize a Trainer using the pretrained model & dataset
trainer = Trainer(
model=model,
train_dataset=dataset["train"],
eval_dataset=dataset["validation"],
)
trainer.train()
trainer.save_model("tomaarsen/span-marker-bert-base-uncased-bionlp-finetuned")
Training Details
Training Set Metrics
| Training set | Min | Median | Max |
|---|
| Sentence length | 2 | 26.5790 | 166 |
| Entities per sentence | 0 | 2.7528 | 23 |
Training Hyperparameters
- learning_rate: 5e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3
Training Results
| Epoch | Step | Validation Loss | Validation Precision | Validation Recall | Validation F1 | Validation Accuracy |
|---|
| 0.4505 | 300 | 0.0210 | 0.7497 | 0.7659 | 0.7577 | 0.9254 |
| 0.9009 | 600 | 0.0162 | 0.8048 | 0.8217 | 0.8131 | 0.9432 |
| 1.3514 | 900 | 0.0154 | 0.8126 | 0.8249 | 0.8187 | 0.9434 |
| 1.8018 | 1200 | 0.0149 | 0.8148 | 0.8451 | 0.8296 | 0.9481 |
| 2.2523 | 1500 | 0.0150 | 0.8297 | 0.8438 | 0.8367 | 0.9501 |
| 2.7027 | 1800 | 0.0145 | 0.8280 | 0.8443 | 0.8361 | 0.9501 |
Environmental Impact
Carbon emissions were measured using CodeCarbon.
- Carbon Emitted: 0.045 kg of CO2
- Hours Used: 0.296 hours
Training Hardware
- On Cloud: No
- GPU Model: 1 x NVIDIA GeForce RTX 3090
- CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
- RAM Size: 31.78 GB
Framework Versions
- Python: 3.9.16
- SpanMarker: 1.3.1.dev
- Transformers : 4.29.2
- PyTorch: 2.0.1+cu118
- Datasets: 2.14.3
- Tokenizers: 0.13.2