VTE-ModernBERT-DeID
VTE-ModernBERT-DeID is a BioClinical ModernBERT-based sequence-classification model for identifying venous thromboembolism (VTE) phenotypes in English clinical narratives. It was fine-tuned on de-identified clinical text and evaluated on the corresponding original clinical text.
Important: “DeID” indicates that de-identified text was used for model fine-tuning. This model classifies VTE phenotypes; it does not perform de-identification.
This checkpoint uses the same underlying annotated dataset, label definitions, and train/test split described for VTE-BERT. The original publication describes the study cohort, clinical annotation process, label development, and validation design.
Model details
- Developed by: Omid Jafari
- Research affiliation and acknowledgement: Ang Li Lab
- Base model: BioClinical ModernBERT large
- Architecture: ModernBertForSequenceClassification
- Task: Single-label text classification
- Language: English
- Architecture context capacity: 8,192 tokens
- Fine-tuning sequence length: 1,024 tokens
- Number of labels: 6
- Release status: Public research release
Although the underlying architecture supports sequences of up to 8,192 tokens, this checkpoint was fine-tuned with a maximum sequence length of 1,024 tokens.
Labels
| ID | Label |
|---|---|
| 0 | LE-DVT+ |
| 1 | PE+ |
| 2 | UE-DVT |
| 3 | atypical |
| 4 | history |
| 5 | none |
Operational definitions for these labels and the annotation procedures are provided in the VTE-BERT publication.
Data preparation and study design
The underlying clinical data and predefined train/test split are the same as those used in the original VTE-BERT study.
- Fine-tuning inputs: Clinical text de-identified with Health-DeID and then prepared for modeling with NLPMed-Engine.
- Test inputs: The corresponding original, non-de-identified clinical text from the held-out test set, prepared with the same NLPMed-Engine preprocessing workflow.
- Distributed data: No training or evaluation data is included in this repository.
Fine-tuning
| Parameter | Value |
|---|---|
| Training text field | final_deidentified_text |
| Maximum training sequence length | 1,024 |
| Architecture context capacity | 8,192 |
| Learning rate | 2e-5 |
| Weight decay | 0.01 |
| Maximum training epochs | 15 |
| Early-stopping patience | 20 |
| Training batch size | 8 |
| Evaluation batch size | 8 |
| Gradient accumulation steps | 1 |
| Model-selection metric | eval_pos_f1 |
| Mixed-precision training | bfloat16 |
The checkpoint was exported and tested with Transformers 5.10.1.
Evaluation
Numerical evaluation results are not included in this model card at this time.
Usage
from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline
model_name = "omid-jf/VTE-ModernBERT-DeID"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
classifier = pipeline(
"text-classification",
model=model,
tokenizer=tokenizer,
)
result = classifier(
"Your clinical text here.",
truncation=True,
max_length=1024,
)
print(result)
For an end-to-end preprocessing and inference workflow, use NLPMed-Engine. Its MLInference component can load Hugging Face sequence-classification checkpoints such as this model.
Intended use and limitations
This model is intended for research involving retrospective VTE phenotyping in clinical narratives, including use cases that may benefit from more context than conventional 512-token BERT models.
It has not been validated for diagnosis, treatment selection, autonomous patient screening, or other clinical decision-making. Performance may vary across institutions, patient populations, note types, documentation practices, preprocessing configurations, and clinical settings. Users are responsible for obtaining appropriate institutional approvals and implementing applicable privacy and security controls.
License
This model is licensed under the GNU Affero General Public License v3.0. The research-use and clinical-safety statements in this model card describe the intended use and known limitations; they do not replace or add restrictions to the AGPLv3 license.
Related resources
- Original gated model: VTE-BERT on Hugging Face
- VTE-BERT publication: Development and Validation of VTE-BERT Natural Language Processing Model for Venous Thromboembolism
- Preprocessing and inference: NLPMed-Engine
- De-identification pipeline: Health-DeID
Citation
For the dataset, annotation process, label definitions, and original VTE-BERT study, please cite:
@article{jafaridevelopment,
title={Development and Validation of VTE-BERT Natural Language Processing Model for Venous Thromboembolism},
author={Jafari, Omid and Ma, Shengling and Lam, Barbara D and Jiang, Jun Y and Zhou, Emily and Ranjan, Mrinal and Ryu, Justine and Bandyo, Raka and Maghsoudi, Arash and Peng, Bo and others},
journal={Journal of Thrombosis and Haemostasis},
publisher={Elsevier},
year={2025},
doi={10.1016/j.jtha.2025.07.021}
}