thealper2/flan-t5-base-medal-abbreviation-expansion
google/flan-t5-base fine-tuned on the MeDAL
pre-training subset for context-aware medical abbreviation expansion: given a
context window and an abbreviation, the model generates the full expansion.
Task format
Input:
Expand the medical abbreviation based on the context.
Context: {context}
Abbreviation: {abbreviation}
Target: the expansion string only, e.g. transverse aortic constriction.
Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("thealper2/flan-t5-base-medal-abbreviation-expansion")
model = AutoModelForSeq2SeqLM.from_pretrained("thealper2/flan-t5-base-medal-abbreviation-expansion")
context = "the patient was admitted to the ICU because of severe respiratory distress"
abbreviation = "ICU"
prompt = (
"Expand the medical abbreviation based on the context.\n\n"
f"Context: {context}\n\n"
f"Abbreviation: {abbreviation}"
)
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=320)
outputs = model.generate(**inputs, max_new_tokens=32, num_beams=4)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training data
MeDAL rows (ABSTRACT_ID, TEXT, LOCATION, LABEL) are converted into
(context, abbreviation, target) triples: the abbreviation is the token at word index
LOCATION in TEXT, the target is LABEL, and the context is a symmetric word window
around LOCATION (MeDAL abstracts are lower-cased and punctuation-free, so there are no
sentence boundaries to cut on).
| Source splits | MeDAL train / valid / test (predefined, not re-mixed) |
| Train examples | 100,000 |
| Validation examples | 10,000 |
| Test examples | 10,000 |
| Context mode | window, 64 words on each side |
| Distinct abbreviations | 5,520 |
| Abbreviations with >1 expansion | 4,391 (79.55%) |
Sub-sampling from the full MeDAL splits uses seeded reservoir sampling; the dataset's own split boundaries are preserved and the test split is used only for the final evaluation.
Training procedure
| Hyper-parameter | Value |
|---|---|
| Base model | google/flan-t5-base |
| Objective | seq2seq cross-entropy |
| Learning rate | 0.0003 |
| LR schedule | linear, warmup ratio 0.03 |
| Batch size | 16 x 2 grad-accum = 32 |
| Epochs | 2.0 |
| Weight decay | 0.01 |
| Max input / target length | 320 / 32 |
| Precision | bf16 |
| Seed | 42 |
| Hardware | NVIDIA GeForce RTX 5060 Ti |
| Runtime | 70.4 min |
| Final validation loss | 0.6663 |
Evaluation
Test split, deterministic decoding (beam search, num_beams=4,
max_new_tokens=32, no sampling).
| Metric | Value (%) |
|---|---|
| Exact Match | 41.87 |
| Normalized Exact Match | 41.88 |
| Token Precision | 53.05 |
| Token Recall | 52.68 |
| Token F1 | 52.58 |
| ROUGE-L | 52.53 |
Ambiguity breakdown
An abbreviation is ambiguous when more than one distinct gold expansion is observed for it in the processed splits.
| Subset | n | Exact Match | Token F1 | ROUGE-L |
|---|---|---|---|---|
| Ambiguous | 9,550 | 42.13 | 52.85 | 52.80 |
| Non-ambiguous | 450 | 36.44 | 46.68 | 46.68 |
Context ablation
Two checkpoints trained with identical data, hyper-parameters and seed; the only difference is whether the prompt contains the context.
| Input formulation | Exact Match | Token F1 | ROUGE-L |
|---|---|---|---|
| Context-aware | 41.87 | 52.58 | 52.53 |
| Context-free | 21.18 | 26.75 | 26.70 |
Limitations
- Trained on PubMed abstracts (MeDAL), not on clinical notes; distribution shift to real clinical text is untested.
- MeDAL labels are produced by automatic reverse substitution, so some gold expansions are noisy or non-canonical; metrics are relative to those labels.
- MeDAL text is lower-cased with punctuation removed; inputs of a different surface form are out of distribution.
- Generative decoding can produce expansions that are not in the label inventory.
- Terminology variants ("heart attack" vs. "myocardial infarction") count as errors under exact match.
- Not a clinical decision-support system and not validated for clinical use.
Citation
@inproceedings{wen-etal-2020-medal,
title = "{M}e{DAL}: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining",
author = "Wen, Zhi and Lu, Xing Han and Reddy, Siva",
booktitle = "Proceedings of the 3rd Clinical Natural Language Processing Workshop",
year = "2020",
pages = "130--135",
url = "https://www.aclweb.org/anthology/2020.clinicalnlp-1.15"
}