CASA-Qwen2_5-VL-3B-LiveCC

This repository contains model weights for CASA-Qwen2_5-VL-3B-LiveCC, a Qwen-2.5VL model adapted from token insertion to a cross-attention-based architecture using CASA layers. It has been further finetuned on the LiveCC dataset for live video captioning.

• Paper: CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
• Project Page: kyutai.org/casa
• Repository: github.com/kyutai-labs/casa

Please refer to the main model card for more information and instructions to run. We provide model weights for other CASA models in the associated model collection.

Quick Start

Below is a short snippet to show you how to load the model, process inputs, and run inference using the 🤗 transformers library. Note that this model requires trust_remote_code=True.

import torch
from transformers.models.auto.modeling_auto import AutoModel
from transformers.models.auto.processing_auto import AutoProcessor

model_id = "kyutai/CASA-Qwen2_5-VL-3B-LiveCC"
model = AutoModel.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).cuda()
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "path/to/your/image.png",
            },
            {
                "type": "text",
                "text": "Describe this image.",
            },
        ],
    },
]
inputs = processor.tokenize_messages(messages=conversation)
inputs = inputs.to(model.device)
input_len = inputs["input_ids"].shape[1]
output_ids = model.generate_from_image(
  **inputs,
  max_new_tokens=512,
  pre_image_tokens=processor.pre_image_tokens,
  post_image_tokens=processor.post_image_tokens,
  eos_token_id=model.generation_config.eos_token_id,
)[0, input_len:]
response = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)

Live video Captioning

In our [github repository], we provide a script to caption a video using our CASA-Qwen2_5-VL-3B-LiveCC model and generate the resulting video with subtitles embedded at the actual time they are generated.

Note that you will also need to install ffmpeg for this script to run. The Python dependencies are handled with uv

# Script options
uv run scripts/gen_livecc_subtitles.py --help
# Generation with Qwen2.5VL+CASA
uv run scripts/gen_livecc_subtitles.py --sample_path path_to_video.mp4 --srt True --temp 0.0

Additional qualitative samples are available in the associated HuggingFace space.

Specifically, video frames are extracted at 2fps, and subtitles are displayed in real-time at the timestamp they are generated<

Citation

If you use CASA in your research, please cite our work:

@article{kyutai2025casa,
  author = {Moritz B\"ohle and Am\'elie Royer and Juliette Marrie and Edouard Grave and Patrick P\'erez},
  year = {2025},
  title = {CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion},
  journal = {ArXiv},
  url = {https://arxiv.org/abs/2512.19535}
}