PathoSynVLM: Case-Level Pathology Synoptic Report Generation
PathoSynVLM is a token-efficient vision-language model for generating case-level pathology synoptic reports from one or more whole-slide images represented as precomputed CONCHv1.5 patch embeddings.
Code and complete documentation: https://github.com/AtlasAnalyticsLab/PathoSynVLM
This repository provides the trained model package. Use it with the PathoSynVLM code repository for embedding preparation, case-level inference, evaluation, and training.
What This Repository Contains
| Path | Purpose |
|---|---|
llm/model.safetensors | Merged language-model weights for the selected Stage 2 checkpoint. |
vlm_state.pt | Vision-language aligner, WSI marker, and WSI index tensors. |
tokenizer/ | Tokenizer configuration and chat template used for inference. |
config.json | PathoSynVLM architecture and inference settings. |
labels.json | Input contract and generated report-field schema. |
best_checkpoint_summary.json | Selected checkpoint and validation summary. |
model_index.json | Machine-readable task, dataset, and metric metadata. |
examples/ | Example case-level input manifest. |
The paper run used unfreeze_llm_base=true, so the release package
includes the merged/full language-model weights under llm/, not only a
LoRA adapter.
Quick Start
The release is loaded through the PathoSynVLM inference code rather than
directly through transformers.AutoModel. A CUDA-capable GPU is
recommended for normal use; CPU execution is intended for smoke tests.
1. Install the code
git clone https://github.com/AtlasAnalyticsLab/PathoSynVLM PathoSynVLM
cd PathoSynVLM
conda create -n pathosynvlm python=3.11 -y
conda activate pathosynvlm
export PYTHONNOUSERSITE=1
pip install -e .
2. Download the model
source configs/paths.example.env
hf download AtlasAnalyticsLab/PathoSynVLM \
--local-dir "$PATHOSYNVLM_WEIGHTS_ROOT/pathosynvlm-stage2-main"
3. Generate a report for one case
Pass every WSI embedding file belonging to the case in the desired slide order:
# Optional when the paths below are relative.
export PATHOSYNVLM_EMBEDDINGS_ROOT=/path/to/conch_v15/embeddings
python scripts/generate_case_report.py \
--embeddings case_001/slide_1.h5 case_001/slide_2.h5 \
--output_json report.json
Relative --embeddings paths are resolved under
PATHOSYNVLM_EMBEDDINGS_ROOT; absolute .h5 paths work without setting
that variable. The JSON output records the generated report, resolved
slide paths, per-WSI patch counts, and feature key.
Generated report text follows:
Diagnosis: ...
Certainty: ...
Conclusion: ...
Input Format
PathoSynVLM runs on precomputed WSI patch embeddings, not raw WSIs. Each
.h5 file should contain:
/features/conch_v15 # shape: (num_patches, 768)
See the GitHub embedding guide for patch extraction, feature generation, H5 validation, and configurable storage paths. Dataset placement and access requirements are documented in the data guide.
From Precomputed H5 Feature Files
This is the fastest path. Put one or more WSI embedding files for a case
into the --embeddings argument:
python scripts/generate_case_report.py \
--embeddings case_slide_1.h5 case_slide_2.h5 \
--output_json report.json
From Raw Whole-Slide Images
First extract tissue patches and CONCHv1.5 patch embeddings using a WSI
preprocessing pipeline that writes the H5 layout above. Then pass the
resulting H5 files to scripts/generate_case_report.py. PathoSynVLM does
not send raw WSI pixels directly to the language model.
Running the Paper Pipeline
Follow the GitHub paper pipeline for the complete sequence: dataset setup, CONCHv1.5 embedding generation, metadata preparation, Stage 1 alignment, Stage 2 case-level fine-tuning, and evaluation. Machine-readable paper configurations and reported values are maintained in the same code repository.
Training Data
The released Stage 2 checkpoint was fine-tuned on case-report pairs from HISTAI. The official metadata repository is the starting point for dataset access and links to the organ-specific WSI repositories used by HISTAI. See the HISTAI source documentation for the dataset structure, subsets, citation, and access instructions.
HISTAI data remain subject to the dataset's CC BY-NC 4.0 license and current access requirements.
Training Recipe
- Stage 1: train the two-layer MLP aligner on HistGen + REG2025 while keeping the CONCHv1.5 patch encoder and LLM frozen.
- Stage 2: fine-tune on HISTAI case-report pairs with WSI marker tokens.
Checkpoint selected for release:
- checkpoint step:
30400 - checkpoint epoch:
7 - validation loss:
1.010892 - prompt style:
double - patch level:
5x_512 - max vision tokens:
4096
Reported Metrics
Stage 1 aligner-only training:
| ROUGE-L | METEOR | BLEU-4 | BERTScore F1 |
|---|---|---|---|
| 0.4743 | 0.4810 | 0.1247 | 0.4253 |
Stage 2 HISTAI main result:
| ROUGE-L | METEOR | BLEU-4 | BERTScore F1 | Diagnosis Exact | Diagnosis Relaxed | Certainty |
|---|---|---|---|---|---|---|
| 0.2495 | 0.1988 | 0.0525 | 0.3018 | 0.1667 | 0.3333 | 0.9000 |
Intended Use
This model is intended for research on pathology report generation from precomputed WSI patch embeddings.
It is not a clinical diagnostic device and should not be used for patient care without appropriate validation, regulatory review, and expert oversight.
License and Commercial Use
This repository uses CC BY-NC-SA 4.0. Research and non-commercial use only. Dataset access, pretrained third-party models, and any externally hosted model weights remain subject to their own terms.
Citation
@inproceedings{yang2026simpletokenvlm,
title = {Simple Token-Efficient Vision-Language Model for Case-Level Pathology Synoptic Report Generation},
author = {Yang, Zhiyuan and Cheng, Jiahao and Trinh, Vincent Quoc-Huy and Hosseini, Mahdi S.},
booktitle = {Proceedings of the 7th International Conference on Deep Learning Theory and Applications},
pages = {514--537},
year = {2026},
issn = {2184-9277}
}