Gaze Attention, Qwen3-4B, video
Video model of the paper Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs (Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, Sangdoo Yun; NAVER AI Lab, KAIST).
- Paper: https://arxiv.org/abs/2605.13080
- Code: https://github.com/junha1125/Gaze-Attention
- Project page: https://june-page.github.io/gaze-attention
Multimodal LLMs attend to all visual tokens at every generation step. With Gaze Attention, the cached keys of the visual tokens are grouped into spatial regions, and every text query attends only to the TopK regions that match it best, selected per layer, per head and per token. A few learnable context tokens per image or frame keep the global view of the scene. The KV cache stays complete, so the attended regions change from one generated token to the next.
Model
| Language model | Qwen3-4B-Instruct-2507 |
| Vision encoder | SigLIP2-So400m/16, 384px (finetuned) |
| Projector | 2-layer MLP with a 3 x 3 convolution (stride 2): 4x fewer visual tokens |
| Visual tokens | 144 per frame |
| Gaze regions | 4 regions of 6 x 6 tokens per frame |
| Attended visual KV entries per query | 10% of the regions of all frames + 4 context tokens per frame (64 frames: 1,192 of 9,216 visual tokens) |
| Context tokens | 4 per frame |
| Parameters | 4.5B, stored in fp16 |
Usage
The model is loaded with the cambrian package of the Gaze Attention repository:
git clone https://github.com/junha1125/Gaze-Attention
cd Gaze-Attention
pip install torch==2.2.2 torchvision==0.17.2
pip install -e .
import copy
import torch
from cambrian.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX
from cambrian.conversation import conv_templates
from cambrian.mm_utils import tokenizer_image_token
from cambrian.model.builder import load_pretrained_model
tokenizer, model, image_processor, _, load_video = load_pretrained_model(
"junha1125/gaze-attention-qwen3-4b-video", None, "cambrian_qwen3", device_map="cuda:0", return_video_loader=True
)
model.eval()
frames, frame_idx, meta = load_video("assets/jobs.mp4", video_fps=1, frames_upbound=64) # 1 FPS, at most 64 frames
video = image_processor.preprocess(frames, return_tensors="pt")["pixel_values"].to(dtype=torch.float16, device="cuda:0")
conv = copy.deepcopy(conv_templates["qwen_3"])
conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nDescribe what happens in this video.")
conv.append_message(conv.roles[1], None)
input_ids = tokenizer_image_token(conv.get_prompt(), tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to("cuda:0")
output_ids = model.generate(input_ids, images=[video], modalities=["video"], do_sample=False, max_new_tokens=256)
print(tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0])
or python demo/video_inference.py --model-path junha1125/gaze-attention-qwen3-4b-video --video assets/jobs.mp4 --max-frames 64.
The model was trained with inputs of up to 12,288 tokens (about 80 frames). For more frames load it with
overwrite_config={"tokenizer_model_max_length": 24576}.
gaze_topk_ratio (0.1 in this checkpoint) is the fraction of gaze regions that a query attends to. It can be changed
when the model is loaded: load_pretrained_model(..., overwrite_config={"gaze_topk_ratio": 0.25}).
The generation config of Qwen3 samples by default; pass do_sample=False for reproducible outputs.
Training
| Stage | Trained parts | Data | Learning rate | Batch size |
|---|---|---|---|---|
| 1-1 | projector | Cambrian-Alignment (1.88M samples) | 1e-3 | 512 |
| 1-2 | projector, context tokens | Cambrian-Alignment | 1e-3 | 512 |
| 2 | all (images, TopK ratio 0.6, then 0.1) | two 17% subsets of Cambrian-7M (1.17M samples each) | 1e-5 (vision encoder 2e-6) | 256 |
| 3 | all (videos, 1 FPS, at most 64 frames) | Cambrian-S-3M subset: 776K video samples + 235K text-only samples | 1e-5 (vision encoder 2e-6) | 256 |
Stage 3 uses the progressive TopK schedule (0.9 to 0.1 during the first 60% of the training steps).
Recipe: scripts/train/video_384 of the repository.
License
Apache License 2.0. The model is built on Qwen3-4B-Instruct-2507 and SigLIP2 (both Apache 2.0). The training datasets have their own licenses.
Citation
@inproceedings{song2026gazeattention,
title={Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal {LLM}s},
author={Junha Song and Byeongho Heo and Geonmo Gu and Jaegul Choo and Dongyoon Han and Sangdoo Yun},
booktitle={Third Conference on Language Modeling (CoLM)},
year={2026}
}