J-Lens for images — reading what a model "thinks" about each part of a picture

When a vision-language model looks at an image, it breaks it into a grid of small squares ("patches") and quietly forms impressions about each one before it writes a single word. This is a tool for reading those impressions. For any patch — or any word the model writes back — it shows the words that patch is disposed to make the model say: a peek at the model's private notes.

It builds on Anthropic's Jacobian lens — a method for reading a language model's internal "workspace" — and extends it, here, to the image side of the vision-language model Qwen3.5-27B. This repo is the fitted lens (jimage_final_lens.pt) plus a short usage script.

The striking thing: the readouts are inferences, not labels. A patch of blank sky can read distant city; a small shape on a roof can read cross → church; a person too small to make out can still read purple (their jacket). Three examples:

🛕 A temple — and inferences beyond it

1. The tower → tower · tall · towering. It also correctly names the building (Meenakshi Amman Temple, Madurai), and nearby patches read temple · Hindu · religious, pyramid (its stepped shape) and even scaffolding / unfinished.
2. Empty sky → city · urban · distant. From an almost featureless hazy patch, the model infers a distant city.
3. A smaller structure beside the tower → smaller · taller · shorter. Not a label — the model is comparing sizes between buildings.

⛪ A church — recognising a symbol, and a tiny detail

1. The cross on the steeple → cross · crucifix. From a small shape it infers the Christian symbol (and, in the readout for the tower below it, church · bell · Christianity).
2. A distant pedestrian → purple — the model reads the colour of the person's jacket, and the patch below reads walking. A fine-grained inference about someone barely a few pixels tall.

🏟️ A stadium — inferring the sport

1. The green field → football · football-field · stadium. From grass and painted lines it infers the sport — and other patches read touchdown · NCAA · seating.

Every readout is verified by the model's own vocabulary head, so these are genuinely the words each patch pushes the model toward — a small window into how it sees.

For the curious — how it works

The lens is a set of matrices J_l (one per layer). Applying J_l to a patch's internal vector and decoding with the model's own output layer gives the readout:

readout_l(h) = softmax( W_U · norm( J_l · h ) ), where J_l = E[ ∂h_final,t' / ∂h_l,t ]

J_l is the average effect a patch's activation has on what the model eventually says. We fit it with image tokens as the source and the text that follows the image as the target — so it captures how each patch shapes the model's answer. (Plain "logit-lens" fails on image tokens; the Jacobian's future-averaging is what makes this work.)

How it was fit

• Model: Qwen/Qwen3.5-27B (bf16, 64 layers, d_model 5120; unified vision-language, early fusion; image placeholder token id 248056).
• Data: 30 single-image windows (retrieval converges from as few as 3; see the paper's corpus-size study) from MMDU-45k (one image + the turn describing it).
• Sources = image-token positions; targets = the answer text. Source layers L16–62 (step 2, 24 layers), target L62, seq 256, bf16 (never quantized — the lens is a derivative on a low-variance subspace, so precision matters).
• Layer note (a real finding): image-source influence ‖J̄_l‖ sits on a broad plateau from L16 through L54 before collapsing (L62 ≈ 0 — it's the target layer), but readout quality peaks at L40–50 (held-out retrieval maximum at L46 — a good default; L50 is equivalent). L62 is a degenerate/empty layer.

Usage

pip install "transformers==5.8.1" accelerate flash-linear-attention pillow \
    git+https://github.com/anthropics/jacobian-lens.git
python apply_example.py your_image.jpg

from jlens.lens import JacobianLens
lens = JacobianLens.load("jimage_final_lens.pt")     # source layers 16..62; use 46
# h = a patch's residual [d] from Qwen3.5's language_model at layer 46
logits = model.lm_head(model.norm(lens.transport(h, 46)))   # top-k words = the readout

apply_example.py (with the bundled mm.py adapter) shows the full path, including mapping an image-token index to its (row, col) patch via the processor's image_grid_thw (patch 16 · spatial-merge 2 → each token ≈ a 32×32-px region).

Files

• jimage_final_lens.pt — the fitted lens (layers 16–62, n=30; ~1.2 GB)
• jimage_L48-62_n120_lens.pt — the original release (layers 48–62, n=120; ~400 MB), kept for reproducibility — per-layer Jacobians are independent, so its layers agree with the full-depth fit
• apply_example.py, mm.py — minimal usage + the multimodal jlens adapter
• examples/ — the highlighted example images above

Notes

• Chinese words appear in some readouts because Qwen3.5's vocabulary is multilingual — the same concept often surfaces in several languages (e.g. city / 城市).

License & lineage

cc-by-nc-4.0, inherited from the MMDU data used to fit the lens (non-commercial). The base model is Apache-2.0; the Jacobian-lens method and jlens library are Apache-2.0 (anthropics/jacobian-lens). Example images are Wikipedia-sourced via MMDU, shown for illustration.

Acknowledgements

Method: Verbalizable Representations Form a Global Workspace in Language Models (Anthropic, 2026). Model: Qwen team. Fitting data: MMDU (Dong, Zhang et al.).