CXR ViT-L/14 — lgc
ViT-L/14 chest-radiograph vision encoder, pretrained from scratch with the objective: Local generative (GCAP/AREF/CAREF box<->phrase) + global image<->report contrastive; the fourth cell of the global/local x generative/contrastive grid.
Part of a controlled study comparing pretraining objectives for chest-X-ray vision encoders (all share the same ViT-L/14 backbone, data, and budget).
Files
| File | What it is |
|---|---|
encoder_final.pt | Vision encoder — ViT-L/14 trunk only (state_dict under key encoder_state, 294 tensors). Use this to extract frozen image features. |
model_best.pt | Full pretraining model — ViT-L/14 + GPT-2-style causal decoder (GCAP/AREF/CAREF) + text encoder + image/text projections + logit_scale (state_dict under key model_state). |
Architecture & training
- ViT-L/14, 384x384 input -> 729 patch tokens (27x27 grid), trained from scratch in bf16. Text side (where present) uses the GPT-2 tokenizer (vocab 50257).
- Training data: MIMIC-CXR (reports) and Chest ImaGenome (region-phrase / anatomy boxes).
- Objective: Local generative (GCAP/AREF/CAREF box<->phrase) + global image<->report contrastive; the fourth cell of the global/local x generative/contrastive grid.
Downstream results (frozen-feature transfer)
Despite a smaller pretraining budget (1 epoch, single A40, vs 8x A100 for the rest — local head strong at that point, val grounding IoU 0.781/acc@0.5 0.895, global contrastive head still near chance R@1 0.063 from a 24-negative pool vs 384 for the others), frozen-feature downstream transfer is consistently top-tier: VQA RadGraph-s 0.231 and BERTScore-F1 0.971 (#1 of every encoder tested, incl. off-the-shelf); Diff-VQA ROUGE-1 0.688 and BERTScore-F1 0.937 (#1); RRG BLEU-4 0.033 (#1 of 10 from-scratch encoders); VinDr-CXR AD mAP@0.4 0.064 / mAP@0.5 0.046 (#3 and #2 of 9 from-scratch, beats every off-the-shelf encoder incl. OWLv2 and SigLIP). Weaker on out-of-domain IU X-ray transfer (RadGraph-s last of 10 after fine-tuning). encoder_final.pt = the epoch-1 best checkpoint (extracted from model_best.pt).
Full per-task comparison across all 9 from-scratch encoders is in the project logbook.
Usage (vision encoder)
import torch
ckpt = torch.load("encoder_final.pt", map_location="cpu", weights_only=False)
vision_state = ckpt["encoder_state"] # 294 tensors, ViT-L/14
# load into your ViT-L/14 implementation, then forward 384x384 images -> [B, 729, 1024]
Intended use & limitations
- Research use only. Frozen feature extraction / fine-tuning for chest-X-ray tasks.
- NOT a diagnostic device. No clinical or patient-facing use.
- Trained only on adult frontal/lateral CXR distributions of the source datasets; may not generalize to other modalities, body regions, populations, or acquisition settings.
⚠️ License / Data Use Agreement
These weights are derived from MIMIC-CXR and Chest ImaGenome, which are distributed under the PhysioNet Credentialed Health Data Use Agreement. Redistribution and use of models derived from these data are subject to that DUA — you must hold the appropriate credentialed access and comply with its terms. Do not make this repository public or share access without confirming your DUA permits sharing derived model weights.
Citation
If you use this encoder, please cite the source datasets (MIMIC-CXR; Chest ImaGenome) and this project.