SkyCaptioner-V1: A Structural Video Captioning Model

Welcome to the SkyCaptioner-V1 repository! Here, you'll find the structural video captioning model weights and inference code for our video captioner that labels the video data efficiently and comprehensively.

🔥🔥🔥 News!!

• Apr 21, 2025: 👋 We release the vllm batch inference code for SkyCaptioner-V1 Model and caption fusion inference code.
• Apr 21, 2025: 👋 We release the first shot-aware video captioning model SkyCaptioner-V1 Model. For more details, please check our paper.

📑 TODO List

• SkyCaptioner-V1

  • Checkpoints
  • Batch Inference Code
  • Caption Fusion Method
  • Web Demo (Gradio)

🌟 Overview

SkyCaptioner-V1 is a structural video captioning model designed to generate high-quality, structural descriptions for video data. It integrates specialized sub-expert models and multimodal large language models (MLLMs) with human annotations to address the limitations of general captioners in capturing professional film-related details. Key aspects include:

1. ​​Structural Representation​: Combines general video descriptions (from MLLMs) with sub-expert captioner (e.g., shot types,shot angles, shot positions, camera motions.) and human annotations.
2. ​​Knowledge Distillation​: Distills expertise from sub-expert captioners into a unified model.
3. ​​Application Flexibility​: Generates dense captions for text-to-video (T2V) and concise prompts for image-to-video (I2V) tasks.

🔑 Key Features

Structural Captioning Framework

Our Video Captioning model captures multi-dimensional details:

• ​​Subjects​: Appearance, action, expression, position, and hierarchical categorization.
• ​​Shot Metadata​: Shot type (e.g., close-up, long shot), shot angle, shot position, camera motion, environment, lighting, etc.

Sub-Expert Integration

• ​​Shot Captioner​: Classifies shot type, angle, and position with high precision.
• ​​Expression Captioner​: Analyzes facial expressions, emotion intensity, and temporal dynamics.
• ​​Camera Motion Captioner​: Tracks 6DoF camera movements and composite motion types,

Training Pipeline

• Trained on ~2M high-quality, concept-balanced videos curated from 10M raw samples.
• Fine-tuned on Qwen2.5-VL-7B-Instruct with a global batch size of 512 across 32 A800 GPUs.
• Optimized using AdamW (learning rate: 1e-5) for 2 epochs.

Dynamic Caption Fusion:

• Adapts output length based on application (T2V/I2V).
• Employs LLM Model to fusion structural fields to get a natural and fluency caption for downstream tasks.

📊 Benchmark Results

SkyCaptioner-V1 demonstrates significant improvements over existing models in key film-specific captioning tasks, particularly in ​shot-language understanding and ​​domain-specific precision​. The differences stem from its structural architecture and expert-guided training:

1. ​​Superior Shot-Language Understanding​:
   • ​Our Captioner model outperforms Qwen2.5-VL-72B with +11.2% in shot type, +16.1% in shot angle, and +50.4% in shot position accuracy. Because SkyCaptioner-V1’s specialized shot classifiers outperform generalist MLLMs, which lack film-domain fine-tuning.
   • ​+28.5% accuracy in camera motion vs. Tarsier2-recap-7B (88.8% vs. 41.5%): Its 6DoF motion analysis and active learning pipeline address ambiguities in composite motions (e.g., tracking + panning) that challenge generic captioners.
2. ​​High domain-specific precision​:
   • ​​Expression accuracy​: ​68.8% vs. 54.3% (Tarsier2-recap-7B), leveraging temporal-aware S2D frameworks to capture dynamic facial changes.

📦 Model Downloads

Our SkyCaptioner-V1 model can be downloaded from SkyCaptioner-V1 Model. We use Qwen2.5-32B-Instruct as our caption fusion model to intelligently combine structured caption fields, producing either dense or sparse final captions depending on application requirements.

# download SkyCaptioner-V1
huggingface-cli download Skywork/SkyCaptioner-V1 --local-dir /path/to/your_local_model_path
# download Qwen2.5-32B-Instruct
huggingface-cli download Qwen/Qwen2.5-32B-Instruct --local-dir /path/to/your_local_model_path2

🛠️ Running Guide

Begin by cloning the repository:

git clone https://github.com/SkyworkAI/SkyReels-V2
cd skycaptioner_v1

Installation Guide for Linux

We recommend Python 3.10 and CUDA version 12.2 for the manual installation.

pip install -r requirements.txt

Running Command

Get Structural Caption by SkyCaptioner-V1

export SkyCaptioner_V1_Model_PATH="/path/to/your_local_model_path"

python scripts/vllm_struct_caption.py \
    --model_path ${SkyCaptioner_V1_Model_PATH} \
    --input_csv "./examples/test.csv" \
    --out_csv "./examepls/test_result.csv" \
    --tp 1 \
    --bs 4

T2V/I2V Caption Fusion by Qwen2.5-32B-Instruct Model

export LLM_MODEL_PATH="/path/to/your_local_model_path2"

python scripts/vllm_fusion_caption.py \
    --model_path ${LLM_MODEL_PATH} \
    --input_csv "./examples/test_result.csv" \
    --out_csv "./examples/test_result_caption.csv" \
    --bs 4 \
    --tp 1 \
    --task t2v

Note:

• If you want to get i2v caption, just change the --task t2v to --task i2v in your Command.

Acknowledgements

We would like to thank the contributors of Qwen2.5-VL, tarsier2 and vllm repositories, for their open research and contributions.

Citation

@misc{chen2025skyreelsv2infinitelengthfilmgenerative,
author = {Guibin Chen and Dixuan Lin and Jiangping Yang and Chunze Lin and Juncheng Zhu and Mingyuan Fan and Hao Zhang and Sheng Chen and Zheng Chen and Chengchen Ma and Weiming Xiong and Wei Wang and Nuo Pang and Kang Kang and Zhiheng Xu and Yuzhe Jin and Yupeng Liang and Yubing Song and Peng Zhao and Boyuan Xu and Di Qiu and Debang Li and Zhengcong Fei and Yang Li and Yahui Zhou},
title = {Skyreels V2:Infinite-Length Film Generative Model},
year = {2025},
eprint={2504.13074},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2504.13074}
}