ChemicalOCR is a compact vision-language model fine-tuned specifically for optical character recognition (OCR) in chemical structure images. It extracts text and bounding boxes from molecular drawings, enabling the recognition of atom labels, abbreviations, and descriptive text within chemical structures.
Overview
ChemicalOCR is fine-tuned from SmolDocling, a lightweight 256M-parameter model originally developed for end-to-end document conversion. By specializing for chemical images, ChemicalOCR substantially outperforms general-purpose OCR systems on chemical structure recognition tasks.
What it does
- Extracts character-level text and bounding boxes from chemical structure images
- Recognizes atom labels, chemical abbreviations, and descriptive text near or below molecular structures
Architecture
- Base model: SmolDocling-256M (Idefics3ForConditionalGeneration)
- Parameters: 256M
- Input: Chemical structure image
- Output: Text detections with bounding boxes
Performance
ChemicalOCR substantially outperforms existing OCR models on chemical structure benchmarks:
| Benchmark | ChemicalOCR (Ours) | PaddleOCR v5 | EasyOCR |
|---|---|---|---|
| M2S (103 images) — F1@IoU₀.₅ | 87.2 | 7.7 | 10.2 |
| USPTO-M (74 images) — F1@IoU₀.₅ | 93.0 | 1.2 | 18.0 |
| IP5-M (1000 images) — F1@IoU₀.₅ | 86.5 | 1.9 | 18.4 |
Detailed Metrics
| Benchmark | Precision | Recall | F1 | Accuracy@IoU₀.₅ |
|---|---|---|---|---|
| M2S | 86.9 | 87.4 | 87.2 | 32.0 |
| USPTO-M | 93.5 | 92.6 | 93.0 | 63.5 |
| IP5-M | 85.6 | 87.4 | 86.5 | 69.5 |
Usage
from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
model = AutoModelForVision2Seq.from_pretrained("docling-project/ChemicalOCR")
processor = AutoProcessor.from_pretrained("docling-project/ChemicalOCR")
image = Image.open("chemical_structure.png")
inputs = processor(images=image, return_tensors="pt")
outputs = model.generate(**inputs)
result = processor.decode(outputs[0], skip_special_tokens=True)
Part of MarkushGrapher-2
ChemicalOCR is a key component of MarkushGrapher-2, an end-to-end model for multimodal recognition of chemical structures from patent documents. In the MarkushGrapher-2 pipeline, ChemicalOCR provides the text and layout information that is fused with visual features for Markush structure recognition.
Citation
@inproceedings{strohmeyer2026markushgrapher2,
title = {MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures},
author = {Strohmeyer, Tim and Morin, Lucas and Meijer, Gerhard Ingmar and Weber, Valery and Nassar, Ahmed and Staar, Peter W. J.},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2026}
}
License
This model is released under the Apache 2.0 License.