DoctoModernBERT-fr-large
📚 Introduction
DoctoModernBERT-fr-large is the large-size French medical encoder in the DoctoBERT family, for biomedical and clinical NLP. It uses the ModernBERT architecture (395M parameters, up to 8192-token context) and is pretrained from scratch on FineMed-fr and FineMed-rephrased-fr.
Its training data is curated from heterogeneous open web corpora (FineWeb-2, FinePDFs, FineWiki), which bring scale, source and stylistic diversity, and inherited quality control. Each document is annotated along three axes: subdomain, educational quality, and medical-term density. DoctoModernBERT then trains on a mix of two parts: the filtered high-quality, entity-rich documents, and LLM-rephrased variants that raise medical-term density and vary the contexts around medical terms across many genres, audiences, and registers. This breadth and density build robustness to real-world clinical text, which is often noisy and inconsistent in style.
For the base-size version, see DoctoModernBERT-fr-base. For a classic RoBERTa encoder, see DoctoBERT-fr-base.
🚀 How to Use
Requires a recent transformers with ModernBERT support.
Fill-mask
Using AutoModelForMaskedLM:
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "doctolib-lab/doctomodernbert-fr-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = f"Le patient souffre d'une {tokenizer.mask_token} aiguë."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, masked_index].argmax(-1)))
Using a pipeline:
from transformers import pipeline
fill = pipeline("fill-mask", model="doctolib-lab/doctomodernbert-fr-large")
print(fill(f"Le patient souffre d'une {fill.tokenizer.mask_token} aiguë."))
For long inputs, load with attn_implementation="flash_attention_2" for faster, more memory-efficient attention.
Fine-tuning
DoctoModernBERT fine-tunes like any BERT/ModernBERT encoder, with the appropriate task head or framework:
- For sequence classification, load it with
AutoModelForSequenceClassification(see the text-classification guide). - For token classification (NER), use
AutoModelForTokenClassification(see the token-classification guide). - For embeddings / retrieval, use Sentence Transformers or PyLate.
📐 Model Overview
| Property | Value |
|---|---|
| Architecture | ModernBERT |
| Parameters | 395M total (343M backbone, 52M embeddings) |
| Layers | 28 |
| Hidden size | 1024 |
| Attention heads | 16 |
| MLP | GeGLU |
| Intermediate size | 2624 |
| Context window | 8192 tokens |
| Vocabulary size | 50,368 |
| Language | French |
🔧 Training
The tokenizer is a SentencePiece BPE model of 50,368 tokens, trained on the entity-rich FineMed-filtered subset (educational quality >= 4 and medical-term density >= 0.1) for efficient tokenization of dense medical vocabulary. It splits digits into single tokens and uses byte fallback for unseen characters.
DoctoModernBERT-fr-large is pretrained from scratch on a mix of FineMed-filtered (FineMed-fr under the same educational-quality and medical-term-density filter) and FineMed-rephrased-fr, over three phases totaling 140B tokens:
- Pretraining (100B tokens). Masked-language-modeling at 1024-token context on the full mix (long documents are hierarchically chunked to fit the context window rather than truncated), building broad medical-language representations.
- Context extension (20B tokens). Extends the context window from 1024 to 8192 tokens, training on a subset upsampled toward long documents.
- Annealing (20B tokens). Continued training on the biomedical & clinical subdomains of the mix, focusing the final updates on content closest to downstream medical use.
⚠️ Intended Use & Limitations
DoctoModernBERT is an encoder for French biomedical and clinical NLP (NER, classification, retrieval, long-document tasks), used by fine-tuning on a downstream task. It is not generative and not a medical device; its outputs must not drive clinical decisions. It was pretrained on public web, PDF, and encyclopedic medical text and reflects the biases and gaps of those sources.
⚖️ License
Released under Apache-2.0. DoctoModernBERT was trained on FineMed-fr and FineMed-rephrased-fr, which derive from FineWeb-2 / FinePDFs (ODC-BY 1.0) and FineWiki (CC BY-SA 4.0); please attribute those upstream sources.
🏛️ Acknowledgments
This work was granted access to the HPC resources of IDRIS (Jean Zay) under the allocations 2025-AD011016291 and 2026-A0200617487 made by GENCI.