Abstract
Video language models process videos as dense visual-token sequences with substantial representational redundancy. Compressing these sequences is therefore essential for reducing the visual-token burden on language-model decoding. The central challenge is to preserve visual information dispersed across frames under such compression. To this end, we introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations onto a compact target measure. The resulting source-to-target coupling induces a distribution over source observations for each target support, directly specifying how the compressed video representation is constructed. We further adapt this construction along task and spatial axes. Question conditioning modulates the transport cost between source frames and target supports, while influencing how many supports are allocated to each temporal segment, thereby directing representation capacity toward question-relevant content. At multiple spatial granularities, AVIOT computes region-specific temporal transport plans and adaptively fuses the representations they yield, allowing different regions within the same compact representation to draw from different moments. Evaluations across varying compression ratios show that AVIOT matches or outperforms the uncompressed baseline on multiple video-understanding benchmarks while retaining strong performance at higher compression ratios.
Model Package
AVIOT is a video-language model for video question answering with
inference-time visual-token compression. This release is a self-contained
inference checkpoint: its four safetensors shards include the language model,
visual encoder, multimodal projector, and AVIOT modules.
The public AVIOT code defines the custom architecture and the video input
pipeline. Install it before loading this checkpoint; generic
AutoModel.from_pretrained() loading is not supported by this release.
Installation
AVIOT requires Python 3.10 or newer, CUDA-enabled PyTorch, and Transformers 4.56 or newer. Clone and install the public code repository:
git clone https://github.com/ernie-research/AVIOT.git
cd AVIOT
python -m pip install -e '.[flash]'
The flash extra installs FlashAttention 2, which is used by the default
inference command. Install the package without that extra and pass
--attn-implementation sdpa if FlashAttention 2 is unavailable.
Loading
Load the checkpoint directly from its Hugging Face repository. The loader downloads and resolves the complete Hub snapshot automatically:
from aviot.inference import AVIOTGenerator
generator = AVIOTGenerator(
"ernie-research/AVIOT",
device="cuda",
torch_dtype="bfloat16",
attn_implementation="flash_attention_2",
)
prediction = generator.answer(
"/path/to/video.mp4",
"What happens in the video?",
ratio=4,
)
print(prediction.text)
The same loader also accepts a downloaded local directory:
from aviot.inference import AVIOTGenerator
generator = AVIOTGenerator(
"/path/to/AVIOT-model",
device="cuda",
torch_dtype="bfloat16",
attn_implementation="flash_attention_2",
local_files_only=True,
)
The command-line interface accepts either the Hub repository ID or a local checkpoint directory:
aviot-infer \
--checkpoint ernie-research/AVIOT \
--video /path/to/video.mp4 \
--question 'What happens in the video?' \
--ratio 4
For a local checkpoint, pass its directory to --checkpoint and add
--local-files-only.
Inference Configuration
The compression ratio controls the target cardinality without requiring a
different checkpoint. For F sampled frames and compression ratio r, AVIOT
constructs ceil(F / r) compact supports (capped at F). The released training
configuration samples ratios from 2 to 10, while --ratio can be selected
directly at inference time.
The default generation configuration is deterministic (do_sample=false). The
CLI likewise defaults to --temperature 0; sampling can be enabled explicitly
by passing a positive temperature.
Video Input
The default video pipeline samples at 2 FPS and retains at most 224 frames.
Frames are resized to 384 by 384 pixels and normalized with the preprocessing
configuration bundled in vision_tower/. The physical frame timestamps are
included in the model prompt.
Citation
If you find AVIOT useful in your research, please cite our paper:
@misc{yin2026aggregatingvisualinformationoptimal,
title={Aggregating Visual Information with Optimal Transport for VideoLM Token Compression},
author={Wenti Yin and Xiaotian Han and Junyuan Shang and Yuchen Ding and Shuohuan Wang and Dianhai Yu and Changxin Gao and Nong Sang},
year={2026},
eprint={2608.20473},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.20473},
}
License
The AVIOT code and this separately distributed checkpoint are released under
the Apache License 2.0. See NOTICE for attribution. Datasets remain subject
to their respective licenses and terms.