SteerViT: Steerable Visual Representations [ECCV'26 Oral]
Paper | Project Page | GitHub | Demo (HF Spaces)
SteerViT equips pretrained Vision Transformers with steerable visual representations. Given an image and a natural-language prompt, it conditions the visual encoder through lightweight gated cross-attention to produce:
- prompt-aware global embeddings
- prompt-aware dense patch features
- prompt-conditioned heatmaps
This Hugging Face repository hosts the model checkpoints.
Installation
To use SteerViT, install the library directly from GitHub:
python -m pip install "git+https://github.com/JonaRuthardt/SteerViT.git"
Quick start
import torch
from PIL import Image
from steervit import SteerViT
device = "cuda" if torch.cuda.is_available() else "cpu"
# Load the model (e.g., SteerDINOv2-Base)
model = SteerViT.from_pretrained("steervit_dinov2_base.pth", device=device)
transform = model.get_transforms()
image = Image.open("path/to/image.jpg").convert("RGB")
image_tensor = transform(image).unsqueeze(0)
prompt = ["the red car"]
global_features = model.get_global_features(image_tensor, texts=prompt) # pooled image embeddings
dense_features = model.get_dense_features(image_tensor, texts=prompt) # patch-level visual features
heatmaps = model.get_heatmaps(image_tensor, texts=prompt) # prompt-conditioned localization heatmaps
attention_heatmaps = model.get_attention_heatmaps(image_tensor, texts=prompt) # attention-based heatmaps
If texts=None, SteerViT behaves like the underlying frozen ViT backbone and returns query-agnostic features.
Available checkpoints
🔔 New commercially usable checkpoints: We now provide updated SteerViT checkpoints under Apache-2.0, which permits commercial use. These variants were adapted using only Visual Genome, COCO, and LVIS, with commercial-use image filtering applied. Thanks to improved synthetic annotations, these variants match or improve upon the original research-only checkpoints.
| Checkpoint | from_pretrained(...) identifier | Notes | Licence |
|---|---|---|---|
| SteerDINOv2-Base | steervit_dinov2_base.pth | Primary model used for most experiments | research-only |
| SteerDINOv2-Small (commercial) | steervit_dinov2_commercial_small.pth | Variant trained on commercially-permissable data | commercial |
| SteerDINOv2-Base (commercial) | steervit_dinov2_commercial_base.pth | Variant trained on commercially-permissable data | commercial |
| SteerDINOv2-Large (commercial) | steervit_dinov2_commercial_large.pth | Variant trained on commercially-permissable data | commercial |
| SteerMAE-Base | steervit_mae_base.pth | Alternative model based on MAE-backbone | research-only |
| SteerSigLIP-Base | steervit_siglip_base.pth | Alternative model based on SigLIP-backbone | research-only |
| SteerSigLIP2-Base | steervit_siglip2_base.pth | Alternative model based on SigLIP2-backbone | research-only |
Citation
@inproceedings{ruthardt2026steervit,
title = {Steerable Visual Representations},
author = {Ruthardt, Jona and Gaur, Manu and Ramanan, Deva and Tapaswi, Makarand and Asano, Yuki M.},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}