CanViT-B, pretrained on ImageNet-21k
CanViT-B pretrained on ImageNet-21k by dense latent distillation from facebook/dinov3-vitb16-pretrain-lvd1689m, as in the paper.
CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.
Paper (NeurIPS 2026) · Code · Project page · All checkpoints
Usage
pip install "canvit-pytorch>=0.2"
import torch
from PIL import Image
from canvit_pytorch import CanViTForPretraining
from canvit_pytorch.episode import run_episode
from canvit_pytorch.policies import make_policy
from canvit_pytorch.preprocess import preprocess
model = CanViTForPretraining.from_pretrained("canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02").eval()
scene = preprocess(512)(Image.open("scene.jpg").convert("RGB")).unsqueeze(0) # [1, 3, 512, 512]
# Five glimpses of 128 px, coarse to fine: the full scene, then quadrants.
policy = make_policy("coarse_to_fine", batch_size=1, device=scene.device, num_glimpses=5, canvas_grid_size=32)
with torch.inference_mode():
steps = run_episode(
canvit=model.canvit, images=scene, policy=policy, num_glimpses=5, glimpse_size_px=128,
initial_state=model.init_state(batch_size=1, canvas_grid_size=32),
)
canvas = steps[-1].state.canvas
features = model.canvit.canvas_patch_grid(canvas) # [1, 32, 32, canvas_dim]: the scene-wide canvas
teacher_patches = model.predict_teacher_patches(canvas) # the teacher's features of the whole scene, standardized
Details
| Pretraining data | ImageNet-21k |
| Teacher | facebook/dinov3-vitb16-pretrain-lvd1689m |
| Scenes | 512 px |
| Glimpses | 128 px |
| Canvas during pretraining | 32 × 32 (any size at inference) |
Citation
@article{berreby2026canvit,
title={CanViT: Toward Active-Vision Foundation Models},
author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
year={2026},
eprint={2603.22570},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.22570}
}
canvit-pytorch 0.1
This repository's files for canvit-pytorch 0.1 remain at revision canvit-pytorch-0.1:
with canvit-pytorch<0.2, pass revision="canvit-pytorch-0.1" to from_pretrained.