FlipLens
A small, MedGemma-faithful chest-X-ray model: a frozen google/medsiglip-448
encoder run at 896 pixels feeding a compact Gemma-3 decoder, with the yes/no
answer read from the tied language-model head. Built to study where paraphrase
sensitivity in medical vision-language models originates.
- Code and the full experiment write-up: github.com/thedatasense/FlipLens
- Interactive write-up: bineshkumar.me/phd-thesis/causality
- Live single-pass flip-detector demo: bineshkumar.me/phd-thesis/explorer/#margin-demo
Scope. A research probe, not a clinical model. It answers binary presence questions about 14 chest findings. Not a medical device; not for clinical use.
What it does
Trained on NIH + PadChest, evaluated on MIMIC and VinDr held out entirely (no image, and no hospital, seen in training). Every split is balanced per finding, so a text-only model scores exactly 0.500 and all accuracy above that is visual.
| split | n | accuracy | AUC | text-only floor | flip rate |
|---|---|---|---|---|---|
| in-distribution (held-out images) | 15,112 | 0.748 | 0.827 | 0.500 | 0.061 |
| MIMIC (unseen hospital) | 450 | 0.671 | 0.743 | 0.500 | 0.049 |
| VinDr (unseen hospital) | 5,478 | 0.686 | 0.756 | 0.500 | 0.060 |
What it establishes about paraphrase sensitivity
Training the same architecture under three phrasing distributions, changing nothing else (8 seeds each; text-only accuracy within one point of chance in all three, 49.4-50.7% across seeds, so none can exploit an answer prior):
| training regime | flip rate | scored on unseen phrasings | accuracy |
|---|---|---|---|
| every paraphrase | 4.8% | 26.6% | 75.3% |
| one fixed phrasing | 67.1% | 65.9% | 66.8% |
| register tied to the answer | 88.4% | 87.4% | 58.1% |
Broad coverage separates from both narrow regimes at Cliff's delta = 1.00 (p = 1.6e-4), and the two narrow regimes separate from each other (p = 3.1e-4, delta = 0.97), so a phrasing shortcut is a distinct and larger harm than narrow coverage alone. The second column trains on half the paraphrase bank and scores only on the 24 phrasings withheld, separating invariance from familiarity with wording already seen: 26.6% against 65.9% is the honest size of the lever.
A rank-1 direction at the answer position restores flipped answers (net recovery 0.98-1.00 across layers 0-4; random-direction control 0.00-0.02; non-flip disruption 0.000-0.006), so the flip is a low-rank, language-side, readout-stage effect. It is not an artifact of where the decision boundary sits: the ordering holds at all 17 offsets across +/-2 SD of the margin, and a threshold-free dispersion statistic reproduces it (0.023 / 0.498 / 1.362, delta = -1.00).
For reference, MedSigLIP's own zero-shot binary performance on this task is AUC 0.734, and an earlier version of this probe scored AUC 0.500 — indistinguishable from blind — on unseen data. Per finding, pneumothorax transfers essentially unchanged (0.899 in-distribution vs 0.903 on VinDr); pleural thickening inverts on VinDr (0.332, below chance), most likely a label-definition mismatch, and should not be trusted.
Accuracy alone cannot tell a seeing model from a blind one here: a blind model, a model given a shuffled grounding token, and a genuinely seeing model all score ~0.50 accuracy while their AUCs are 0.500 / 0.502 / 0.604. Report AUC.
The discarded margin is a single-pass flip detector
Because the answer is the sign of the yes-minus-no margin and a flip is a sign change,
a margin near zero marks a paraphrase-unstable answer. The absolute margin of one
forward pass therefore ranks flip-prone questions at no cost beyond the answer itself,
and it beats the detectors that cost more (scripts/analysis/detect.py):
| detector | passes | in-distribution | MIMIC | VinDr |
|---|---|---|---|---|
| absolute margin | 1 | 0.923 | 0.974 | 0.974 |
| paraphrase self-consistency | k | 0.709 | 0.786 | 0.624 |
| hidden-state probe | 1 + fit | 0.826 | 0.838 | 0.801 |
The other failure, an answer that ignores the image, has no single-pass signal: the margin is at chance for it (0.470 to 0.519), because a confident blind answer looks exactly like a confident grounded one. Only a second pass that swaps in another patient's image detects it (0.827 to 0.907). A finer question, whether a stable answer quietly changes how much it uses the image from one phrasing to another, is real at the population level (cross-draw covariance 0.062, between-draw correlation 0.56, 82% case-specific over three seeds) but not identifiable per case (zero grounded-to-unreliant transitions over 345 patient-seed evaluations, patient-level upper bound near 3%). Image reliance is a population property here, not a per-prediction label. Try it in the live margin-gate demo.
Load
import torch
from transformers import AutoModel
from PIL import Image
model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True).eval()
input_ids, ans_pos = model.encode_question("is there pleural effusion?")
vision_features, ground = model.encode_images([Image.open("cxr.png").convert("RGB")])
logits = model(input_ids=input_ids, vision_features=vision_features,
ground=ground, ans_pos=ans_pos).logits
print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
encode_images needs gated access to google/medsiglip-448.
Repository structure
config.json model.safetensors modeling_babymedgemma.py the model + self-contained wrapper
Training and analysis code, and the exact scripts that produced every number here, are on GitHub: thedatasense/FlipLens.
Citation
Sadanandan, B. (2026). Paraphrase Sensitivity in Medical Vision-Language Models: Measurement, Mechanisms, Mitigation, and Deployment Safety. PhD dissertation, University of New Haven.