MultiClinicalBERT: A Multilingual Transformer Pretrained on Real-World Clinical Notes
MultiClinicalBERT is a multilingual transformer model pretrained on real-world clinical notes across multiple languages. It is designed to provide strong and consistent performance for clinical NLP tasks in both high-resource and low-resource settings.
To the best of our knowledge, this is the first open-source BERT model pretrained specifically on multilingual real-world clinical notes.
Model Overview
MultiClinicalBERT is initialized from mmBERT and further pretrained using a two-stage domain-adaptive strategy on a large-scale multilingual clinical corpus (BRIDGE), combined with biomedical and general-domain data.
The model captures:
- Clinical terminology and documentation patterns
- Cross-lingual representations for medical text
- Robust performance across diverse healthcare datasets
Pretraining Data
The model is trained on a mixture of three data sources:
1. Clinical Data (BRIDGE Corpus)
- 87 multilingual clinical datasets
- ~1.42M documents
- ~995M tokens
- Languages: English, Chinese, Spanish, Japanese, Russian
This dataset reflects real-world clinical practice and is the core contribution of this work.
2. Biomedical Literature (PubMed)
- ~1.25M documents
- ~194M tokens
Provides domain knowledge and medical terminology.
3. General-Domain Text (Wikipedia)
- ~5.8K documents
- ~43M tokens
- Languages: Spanish, Japanese, Russian
Improves general linguistic coverage.
Total
- ~2.7M documents >1.2B tokens
Pretraining Strategy
We adopt a two-stage domain-adaptive pretraining approach:
Stage 1: Mixed-domain pretraining
- Data: BRIDGE + PubMed + Wikipedia
- Goal: Inject biomedical and multilingual knowledge
Stage 2: Clinical-specific adaptation
- Data: BRIDGE only
- Goal: Learn fine-grained clinical language patterns
Objective
- Masked Language Modeling (MLM)
- 15% token masking
Evaluation
We evaluate MultiClinicalBERT on 10 clinical NLP tasks across 5 languages:
- English: MIMIC-III Mortality, MIMIC-IV CDM
- Chinese: CEMR, IMCS-V2 NER
- Japanese: IFMIR Incident Type, IFMIR NER
- Russian: RuMedNLI, RuCCoN NER
- Spanish: EHR De-identification, PPTS
Key Results
- Consistently outperforms Modern multilingual BERT (mmBERT)
- Matches or exceeds strong language-specific models
- Largest gains observed in low-resource settings
Example:
- MIMIC-III Mortality: 89.40% accuracy
- MIMIC-IV CDM: 92.18% accuracy
- CEMR: 94.24% accuracy
- IFMIR NER: 86.21 F1
- EHR De-identification: 88.64 F1
Key Contributions
- First BERT model pretrained on multilingual real-world clinical notes
- Large-scale clinical corpus (BRIDGE) with diverse languages
- Effective two-stage domain adaptation strategy
- Strong performance across multiple languages and tasks
- Suitable for:
- Clinical NLP
- Multilingual medical text understanding
- Retrieval-augmented generation (RAG)
- Clinical decision support systems
Usage
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("YLab-Open/MultiClinicalBERT")
model = AutoModel.from_pretrained("YLab-Open/MultiClinicalBERT")