TransVLM-Qwen3-VL-4B-Instruct

[!IMPORTANT] This is not a drop-in Qwen3-VL checkpoint. Its vision tower takes 6 channels: RGB concatenated with an optical-flow visualisation (config.json → vision_config.in_channels: 6). A standard AutoProcessor / Qwen3VLForConditionalGeneration pipeline supplies 3 channels and will not produce correct results. The weights load, but the input is wrong.

Run it through the released inference code: https://github.com/heygen-com/TransVLM

Model Description

Traditional Shot Boundary Detection (SBD) looks for isolated cut points, which breaks down on gradual transitions and frequently yields corrupted shots. TransVLM targets Shot Transition Detection (STD) instead: it predicts the continuous temporal segments over which a transition happens.

The model is Qwen3-VL-4B-Instruct with one architectural change. Its vision patch embedding is widened from 3 to 6 input channels (zero-padded at initialisation), so an optical-flow visualisation enters the model alongside RGB at the input stage. Motion is what separates a dissolve from a camera pan, and a model that only sees appearance has to infer it. Because the flow is fused before patchification, the language backbone carries no additional visual tokens, so temporal awareness comes for free at the token budget.

Training used a scalable FFmpeg-based data engine covering 59 transition effects, which addresses the severe class imbalance in public shot-boundary data.

Model Details

Intended Use and Limitations

Intended for detecting shot transitions in videos, both hard cuts and gradual effects, and as a baseline on the STD benchmark.

Limitations

• Not a frame-level cut-point classifier. The output is a time span per transition, not a single boundary frame.
• Not a general video-chat model. It was fine-tuned on one task with one prompt; the prompt ships with the inference code and changing it changes the task.
• Long videos should be split before inference. Cost is linear in duration, and the flow visualisation is normalised over its whole input, so a long video and its parts are not the same signal.

Usage

No standalone transformers snippet is given here on purpose: a 3-channel one would run and return nonsense. Use the inference package, which needs Python 3.12, a CUDA GPU, and ffmpeg on PATH:

git clone https://github.com/heygen-com/TransVLM
cd TransVLM/inference

uv venv && source .venv/bin/activate
uv sync --group cu130 --group dev        # cu128 if your driver is older than 570

# Do not skip this. PyTorch 2.9.1 has a Conv3d bug below cuDNN 9.15, and the 6-channel
# patch embed IS a Conv3d, so it is on the hot path of every forward pass. `uv sync`
# resolves cuDNN back down, so re-run this after every sync. Activate the venv first:
# unlike `uv sync`, `uv pip` installs into whatever environment is active.
uv pip install nvidia-cudnn-cu13==9.16.0.29        # nvidia-cudnn-cu12 for cu128 / cu126
python -c "import torch; print(torch.backends.cudnn.version())"   # must print 91600

hf download HeyGenAI/TransVLM-Qwen3-VL-4B-Instruct --local-dir ./pretrained/TransVLM-v1

python infer_video.py \
  --video /path/to/video.mp4 \
  --ckpt-dir ./pretrained/TransVLM-v1 \
  --output-jsonl out.jsonl

The cuDNN package name tracks the CUDA flavour, and installing the wrong one is a silent no-op: it lands an unused package while torch keeps loading the other family. That is why the check prints a version number instead of trusting the install.

Each output line carries segments as {start_time, end_time} pairs in seconds on the original video's timeline, plus timings and the full configuration that produced them.

Three backends are supported: HuggingFace (default), vLLM and SGLang. Environment setup, every option and the output schema are documented in inference/README.md.

Evaluation

Qualitative results and the comparison against baselines are on the project page.

The scorer that produced the paper's metric is released at evaluation/. Note that the published figures cannot be reproduced bit-for-bit from the released artifacts alone. The benchmark ground truth and the pre-computed optical flow used for that run are not part of this release. The reasons are spelled out in evaluation/README.md.

Release Progress

• Model weights
• Inference code
• Evaluation code
• Data engine code
• STD benchmark data
• Re-annotated dataset labels
• Leaderboard

🚧 The remaining items are being prepared for release.

Citation

@inproceedings{chen2026transvlm,
  title={TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions},
  author={Chen, Ce and Ren, Yi and Li, Yuanming and Goriachko, Viktor and
          Ye, Zhenhui and Guo, Zujin and Hong, Zhibin and Gong, Mingming},
  booktitle={European Conference on Computer Vision},
  year={2026},
  organization={Springer}
}

License

Apache License 2.0, inherited from Qwen3-VL-4B-Instruct.

Scope of this release

TransVLM was developed as academic research and is published at ECCV 2026. It is not used in any HeyGen product; HeyGen's production system for this task is a different model, trained separately from this one.

Training data

Training combined HeyGen's own generated and annotated footage, which accounts for the majority of the sampling distribution, with four public shot-boundary datasets: AutoShot, ClipShots, MovieShots2 (SCTSNet, whose cut annotations build on the Condensed Movies Dataset), and SportsShot. Together these four are roughly 42% of the training sampling distribution, of which SportsShot is about 7.5%.

SportsShot is licensed CC BY-NC 4.0 and is credited here accordingly. The other three distribute annotations rather than footage, so consult each dataset's own terms before relying on them. No dataset is redistributed by this repository, and no dataset footage is contained in these weights.