Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities
Muhammad Uzair Khattak, Won Jun Kim, Reza Abbassi, Albias Havolli, Michael Murphy, Amir Zadeh, Chuan Li, Muhammad Ferjad Naeem*, Oğuzhan Fatih Kar*, Roman Bachmann*, Andrei Atanov*, Federico Tombari, Amir Zamir
EPFL, Google, Lambda
Official pre-trained checkpoints for Video-4M, an any-to-any multimodal model for the video domain that allows flexible traversal across both modality and time axes.
🌐 Project page | 📄 Paper | 💻 Code | 🤗 Tokenizers | 🤗 Demo | BibTeX
Model checkpoints
This repository hosts the main Video-4M model checkpoints. We release two checkpoint versions, both pretrained on the K600-MM dataset:
| Model | Params | File | Config |
|---|---|---|---|
| Video-4M-L | 705M | main_model/Video-4M-L.pth | Config |
| Video-4M-Pred-L | 705M | main_model/Video-4M-Pred-L.pth | Config |
Video-4M-Pred is a Video-4M model whose pretraining input-output mixture is biased toward predicting video from sparse modalities (e.g., caption, transcription) and toward forecasting tasks. It can still perform all other any-to-any tasks.
Usage
We provide two ways to get started with Video-4M: the 🤗 Space demo, which lets you test Video-4M without any local setup, and the notebooks in our main GitHub code repository. Both download the model checkpoints and the tokenizers automatically.
Below, we provide a minimal example of performing inference with Video-4M using our notebook pipeline. Here, we give an RGB video as input and generate all other modalities via chained generation:
from notebooks import pipeline
from notebooks.display_utils import video_grid
tokenizers = pipeline.load_tokenizers() # downloads the tokenizers from the Hub on first call
model = pipeline.load_model(tokenizers, model_name="Video-4M_Pred_L") # downloads the model from the Hub on first call
example = pipeline.load_example(pipeline.list_examples()[0]) # downloads pre-tokenized video examples from the Hub, and selects the first one
results = pipeline.generate(
example, input_modalities=["rgb"],
chain=pipeline.CHAIN_PRESETS["rgb_to_others"]["chain"], # you can also define a custom chain, e.g. ['caption', 'depth']
tokenizers=tokenizers, model=model,
)
video_grid(results, cols=5, width=170, title="rgb -> everything else") # for visualizing the generation results
For installation, training, and evaluation instructions, please see the README of our main GitHub code repository.
License
The model weights are released under the Sample Code license as found in LICENSE_WEIGHTS.
Citation
If you find this repository useful, please consider citing:
@article{video4m2026,
title={{Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities}},
author={Khattak, Muhammad Uzair and Kim, Won Jun and Abbassi, Reza and Havolli, Albias and Murphy, Michael and Zadeh, Amir and Li, Chuan and Naeem, Muhammad Ferjad and Kar, O\u{g}uzhan Fatih and Bachmann, Roman and Atanov, Andrei and Tombari, Federico and Zamir, Amir},
journal={arXiv preprint},
year={2026},
}