CamemBERT-bio-v2-base
CamemBERT-bio-v2-base is a French biomedical encoder for token-level tasks. It is obtained by continuing the pretraining of camembertv2-base on biomedical text. On our named entity recognition benchmark it reaches 65.8 exact-F1, among the best results we measured for a French biomedical encoder of this size. For long-document tasks, see ModernCamemBERT-bio-v2-large.
Authors
Rian Touchent and Eric de la Clergerie, Sorbonne Université, Inria Paris.
Overview
This model targets clinical and biomedical entity recognition in French. Starting from camembertv2-base, we first run a causal language modeling detour on clinical text, then a decay phase that masks whole entity spans and gives more weight to medical terms. The detour deepens the domain adaptation, and the entity-focused masking sharpens the token representations that entity recognition relies on.
| Architecture | RoBERTa (CamemBERTv2) |
| Parameters | 111M |
| Context length | 1,024 tokens |
| Language | French |
| Base model | almanach/camembertv2-base |
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "rntc/CamemBERT-bio-v2-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = "Le patient présente une <mask> aiguë du myocarde."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits
i = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, i].argmax(-1)))
Training
The model is initialized from camembertv2-base. We first apply a causal language modeling detour on the MC-Bio clinical corpus, which produces lasting changes in the early layers and improves domain adaptation. We then run a decay phase with whole-entity-span masking, where medical surface forms are masked as full spans and sampled more often than ordinary words. This keeps the masked language modeling objective throughout.
Evaluation
Per-task results on the short named entity recognition core of our French biomedical benchmark, scored with exact-F1. Our models are averaged over nine seeds; the external baselines are currently averaged over three seeds.
| Model | EMEA | Medline | E3C | Avg |
|---|---|---|---|---|
| DoctoBERT | 72.0 | 66.5 | 61.5 | 66.7 |
| CamemBERT-bio-v2 | 71.9 | 64.7 | 60.9 | 65.8 |
| CamemBERT-bio | 70.6 | 65.1 | 61.5 | 65.7 |
| DoctoModernBERT | 68.7 | 65.4 | 60.7 | 64.9 |
| DrBERT | 69.0 | 62.9 | 58.5 | 63.4 |
| ModernCamemBERT-bio-v2-large | 67.8 | 63.4 | 58.7 | 63.3 |
| ModernCamemBERT-bio-base | 67.6 | 60.7 | 58.7 | 62.3 |
| ModernCamemBERT-bio-v2-base | 64.0 | 59.6 | 58.5 | 60.7 |
CamemBERT-bio-v2 comes second on the average, close behind doctobert and ahead of DoctoModernBERT. Most of the remaining gap comes from Medline, the scientific-title subset.
Environmental impact
The biomedical continued-pretraining reported here ran for about 1 GPU-hour on H100, on the Jean Zay cluster (GENCI-IDRIS) in France. We estimate roughly 0.03 kg CO2eq for this phase. This figure covers the continued-pretraining step only and does not include the pretraining of the base model.
License
MIT
Citation
@inproceedings{touchent:hal-05697506,
TITLE = {{OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining}},
AUTHOR = {Touchent, Rian and de la Clergerie, {\'E}ric},
URL = {https://hal.science/hal-05697506},
BOOKTITLE = {{Proceedings of Knowledge Graphs and Large Language Models Workshop}},
ADDRESS = {Palma de Mallorca, Spain},
YEAR = {2026},
MONTH = May,
PDF = {https://hal.science/hal-05697506v1/file/main.pdf},
HAL_ID = {hal-05697506},
HAL_VERSION = {v1},
}
@misc{touchent2026causallanguagemodelingdetour,
title={A Causal Language Modeling Detour Improves Encoder Continued Pretraining},
author={Rian Touchent and Eric de la Clergerie},
year={2026},
eprint={2605.12438},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.12438},
}