Dispider

Official checkpoint for Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction (CVPR 2025).

• Code
• Paper

Dispider continuously perceives a video stream, uses a compact Decision model to determine when a response is needed, and invokes the larger Reaction model only after a trigger.

Architecture

Checkpoint layout

The composite shards contain all model weights exactly once. The nested perception_decision/ directory contains metadata only.

Dispider/
├── config.json
├── dispider_checkpoint_manifest.json
├── model-00001-of-00004.safetensors
├── model-00002-of-00004.safetensors
├── model-00003-of-00004.safetensors
├── model-00004-of-00004.safetensors
├── model.safetensors.index.json
└── perception_decision/
    ├── config.json
    ├── tokenizer metadata
    └── vision_tower/
        ├── config.json
        └── preprocessor_config.json

Download it directly:

huggingface-cli download Mar2Ding/Dispider \
  --local-dir checkpoints/Dispider

Use the code repository with Python 3.10, PyTorch 2.2.0, FlashAttention 2.5.9.post1, and Transformers 4.41.2. Load either the repository ID or a local snapshot through dispider.model.load_pretrained_model; generic AutoModel.from_pretrained() does not assemble the video components. The accompanying code is an inference-only runtime.

Offline inference

CUDA_VISIBLE_DEVICES=0 python inference.py \
  --model_path Mar2Ding/Dispider \
  --video_path /path/to/video.mp4 \
  --prompt "What is happening in the video?"

Streaming inference

CUDA_VISIBLE_DEVICES=0 python stream_inference.py /path/to/video.mp4 \
  --model Mar2Ding/Dispider \
  --prompt "Tell me when the person starts performing." \
  --decision-kv-cache auto

The command emits timestamped JSON responses as Decision triggers Reaction. The optional Decision KV cache stores only stable prefixes and automatically falls back to full computation near the trigger threshold.

OVO-Bench example

From the code repository:

python scripts/evaluate_ovobench.py \
  --ovo-root ../OVO-Bench \
  --model-path checkpoints/Dispider \
  --data-root /path/to/ovo-data \
  --output-dir outputs/ovobench \
  --gpus 0,1,2,3 \
  --resume

The evaluator validates complete coverage, rejects missing or null outputs, and computes the official category metrics. Generated results remain local.

Citation

@inproceedings{qian2025dispider,
  title={Dispider: Enabling Video {LLMs} with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction},
  author={Qian, Rui and Ding, Shuangrui and Dong, Xiaoyi and Zhang, Pan and Zang, Yuhang and Cao, Yuhang and Lin, Dahua and Wang, Jiaqi},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
  pages={24045--24055},
  month={June},
  year={2025}
}