MedSigLIP Radiology Modality Classifier (v3) — with OOD Gate

Fine-tuned from google/medsiglip-448 for zero-shot image classification across 17 classes — 11 radiology modalities plus 6 non-radiology categories. v3 adds a binary radiology gate that rejects non-radiological images before classification, achieving AUROC 0.910 for out-of-distribution detection.

Model Description

This is a contrastive fine-tune of MedSigLIP — both vision and text encoders were trained jointly with SigLIP loss on image+caption pairs. Unlike classification-head approaches that freeze the vision encoder, this preserves and strengthens MedSigLIP's vision-language alignment.

v3 extends v2 by adding 6 non-radiology classes (Clinical Photograph, Gross Pathology Specimen, Pathology Histology, Schematic/Illustration, DEXA, PET-MRI) to the 11 radiology classes from v2. This enables the model to not only classify radiology modalities but also reject non-radiological images via a dedicated binary gate.

Classes (17 total)

Radiology (11): Angiography, CT, Fluoroscopy, Mammography, MRI, PET, PET-CT, Scintigraphy, SPECT, Ultrasound, X-ray

Non-Radiology (6): Clinical Photograph, DEXA, Gross Pathology Specimen, Pathology Histology, PET-MRI, Schematic / Illustration

Key Innovation: Binary Radiology Gate

v3 includes a lightweight binary classifier gate (nn.Linear(1152, 1) + sigmoid) trained on top of the frozen vision encoder's pooler_output. The gate learns a hyperplane that separates radiology visual features from non-radiology ones:

Why the gate works (and other OOD methods failed):

• Energy-based OOD (AUROC 0.581): Contrastive loss produces similar logit magnitudes for all classes — no separation.
• Nearest-centroid OOD (AUROC 0.522): All 17 class centroids are within ~0.98 cosine similarity on the hypersphere; every image finds a nearby centroid.
• Linear gate (AUROC 0.910): Learns a dedicated hyperplane that separates radiology visual features (grids, grayscale, anatomical structure) from non-radiology ones — without relying on inter-class separation.

The gate is stored as a standalone JSON file (radiology-gate.json, ~30KB) with weights, bias, threshold, and class metadata — no PyTorch dependency needed for loading.

Pipeline

Image → Vision Encoder → pooler_output (1152-dim)
                              │
                              ├─→ Binary Gate: sigmoid(W·x + b) → score
                              │       score < 0.868 → REJECT (non-radiology)
                              │       score ≥ 0.868 → ACCEPT
                              │
                              └─→ L2 normalize → cosine similarity with text prompts
                                      → softmax → top-k modality prediction

Training

Training Data

~5,000+ images across 17 classes (11 radiology + 6 non-radiology), each paired with a modality-specific text prompt.

Class prompts:

Training Procedure

• Loss: SigLIP contrastive loss (image-text alignment within batch)
• Epochs: 10
• Batch size: 32
• Learning rate: 1e-4 with cosine schedule
• Optimizer: AdamW (weight decay 0.01)
• Precision: FP16 mixed (on GPU)
• Best checkpoint: 1755
• Weighted sampling: Inverse class-frequency oversampling

Hardware

Trained on a single RTX Pro 6000.

How to Use

Basic Zero-Shot Classification

from transformers import AutoModel, AutoProcessor
from PIL import Image
import torch

model = AutoModel.from_pretrained("kashol/medsiglip-modality-ft-v3")
processor = AutoProcessor.from_pretrained("kashol/medsiglip-modality-ft-v3")

MODALITY_PROMPTS = {
    "Angiography":              "an angiography image",
    "CT":                       "a CT scan",
    "Clinical Photograph":      "a clinical photograph",
    "DEXA":                     "a DEXA bone density scan",
    "Fluoroscopy":              "a fluoroscopy image",
    "Gross Pathology Specimen": "a gross pathology specimen",
    "Mammography":              "a mammogram",
    "MRI":                      "an MRI scan",
    "PET":                      "a PET scan",
    "PET-CT":                   "a PET-CT scan",
    "PET-MRI":                  "a PET-MRI scan",
    "Pathology Histology":      "a photomicrograph or histology slide",
    "Schematic / Illustration": "a schematic or diagram",
    "Scintigraphy":             "a scintigraphy or nuclear medicine scan",
    "SPECT":                    "a SPECT scan",
    "Ultrasound":               "an ultrasound image",
    "X-ray":                    "an X-ray radiograph",
}

modalities = list(MODALITY_PROMPTS.keys())
prompts = list(MODALITY_PROMPTS.values())

image = Image.open("chest_ct.jpg").convert("RGB")
text_inputs = processor(text=prompts, padding="max_length",
                        truncation=True, max_length=64, return_tensors="pt")
img_inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(
        pixel_values=img_inputs["pixel_values"],
        input_ids=text_inputs["input_ids"],
        attention_mask=(text_inputs["input_ids"] != 0).long(),
    )
    pred_idx = outputs.logits_per_image.argmax(dim=-1).item()

print(f"Predicted: {modalities[pred_idx]}")

With Binary Radiology Gate

For production use where non-radiology images must be rejected, load the gate alongside the model:

import json
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoProcessor
from PIL import Image

# Load gate
with open("models/radiology-gate.json") as f:
    gate = json.load(f)

gate_weight = torch.tensor(gate["weights"]).float()       # (1152,)
gate_bias = torch.tensor(gate["bias"]).float()
gate_threshold = gate["threshold"]  # 0.868

# Load model
model = AutoModel.from_pretrained("kashol/medsiglip-modality-ft-v3")
processor = AutoProcessor.from_pretrained("kashol/medsiglip-modality-ft-v3")

image = Image.open("some_image.jpg").convert("RGB")
img_inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    # Get raw pooler_output (NOT L2-normalized)
    vision_outputs = model.vision_model(
        pixel_values=img_inputs["pixel_values"]
    )
    pooler = vision_outputs.pooler_output  # (1, 1152)

    # Gate check
    gate_score = torch.sigmoid(gate_weight @ pooler[0] + gate_bias).item()

    if gate_score < gate_threshold:
        print(f"REJECTED — not a radiology image (gate score: {gate_score:.3f})")
    else:
        # Proceed with zero-shot classification
        text_inputs = processor(text=MODALITY_PROMPTS.values(), ...)
        outputs = model(
            pixel_values=img_inputs["pixel_values"],
            input_ids=text_inputs["input_ids"],
            attention_mask=(text_inputs["input_ids"] != 0).long(),
        )
        pred = modalities[outputs.logits_per_image.argmax(dim=-1).item()]
        print(f"ACCEPTED (gate: {gate_score:.3f}) → {pred}")

classify_server.py (FastAPI Microservice)

A production-ready FastAPI server is included in the repository:

python classify_server.py \
  --model kashol/medsiglip-modality-ft-v3 \
  --gate models/radiology-gate.json \
  --host 0.0.0.0 --port 8765

Endpoints:

• POST /classify — Full pipeline: image → gate check → zero-shot classification
• POST /embed — Return raw 1152-dim pooler_output embedding
• GET /classify/health — Gate status, AUROC, threshold, class list

Example response from /classify:

{
  "is_radiology": true,
  "gate_score": 0.952,
  "gate_threshold": 0.868,
  "gate_auroc": 0.910,
  "modality": "CT",
  "all_classes": [
    {"label": "CT", "score": 0.987},
    {"label": "MRI", "score": 0.008},
    {"label": "PET-CT", "score": 0.003}
  ],
  "note": "Gate passed. Top-3 zero-shot classification results."
}

Rejected non-radiology image:

{
  "is_radiology": false,
  "gate_score": 0.321,
  "gate_threshold": 0.868,
  "gate_auroc": 0.910,
  "modality": null,
  "all_classes": [],
  "note": "Image rejected by radiology gate (score 0.321 < threshold 0.868)."
}

OOD Detection: Three Methods Compared

Three out-of-distribution detection methods were evaluated on the same test set. Only the binary gate proved effective:

Why contrastive embedding space fails for centroid/energy OOD: All 17 class centroids are within ~0.98 cosine similarity of each other. The embedding space is a tight ball on the 1152-dim hypersphere — there is no "empty region" for unknown images to fall into. The linear gate works because it learns a hyperplane rather than relying on distance to centroids.

Limitations

• Schematics & Histology false positives: Schematics and pathology histology images score at P=1.000 on the gate (17.8% of non-radiology images pass). These share visual features with radiology (grids, grayscale, structured content). Mitigation: more training data or multi-label heads.
• Nuclear medicine confusion: SPECT, Scintigraphy, and PET share visual characteristics and are the primary source of confusion among radiology classes.
• Single-modality assumption: The model classifies each image as exactly one class.
• Not for diagnosis: This model identifies imaging modality, NOT pathology. It is a triage, search, and organization tool — not a diagnostic aid.

Files

Citation

@misc{medsiglip-modality-ft-v3-2026,
  author = {kashol},
  title = {MedSigLIP Radiology Modality Classifier v3 — with OOD Gate},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/kashol/medsiglip-modality-ft-v3}},
}

Base model: google/medsiglip-448