video-ma2mba-0.3b
Video-Ma²mba is an efficient long-form video understanding model that replaces Transformer attention with State Space Models (SSMs) in the Mamba-2 framework, achieving linear time and memory scaling in sequence length. A single GPU can therefore process video sequences equivalent to millions of tokens — over two hours of video at 1 FPS.
Paper: Look Every Frame All at Once: Video-Ma²mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing
Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro
Integrated Vision and Language Lab, KAIST
What is Video-Ma²mba?
Processing long videos with Transformer-based video-LLMs is expensive: self-attention scales quadratically with the number of tokens, so memory and compute explode as videos grow to thousands of frames. Video-Ma²mba addresses this at the architecture level.
- State-space backbone. The language model is built on Mamba-2, replacing attention with selective state-space layers whose cost grows linearly with sequence length. A long video is consumed as one continuous stream rather than a truncated window.
- Multi-Axis Gradient Checkpointing (MA-GC). A training-time memory technique that checkpoints activations along multiple axes, sharply reducing the memory footprint versus standard gradient checkpointing and making long-sequence training feasible.
- Streaming-scale context. Together these let the model ingest sequences equivalent to millions of tokens — 2+ hours of video at 1 FPS — on a single GPU.
Architecture
Video (sampled at 1 FPS)
│
▼
[SigLIP-so400m Vision Encoder] → per-frame visual tokens
│
▼
[MLP Projector] → language-model embedding space
│
▼
[Mamba-2 Language Model] ← linear-time SSM · 48 layers · d=1024
│
▼
Text response
| Component | Specification |
|---|---|
| Architecture | LlavaMambaForCausalLM (llava_mamba) |
| Language model | Mamba-2 · 48 layers · hidden size 1024 · ~370M params |
| Vision encoder | google/siglip-so400m-patch14-384 |
| Projector | 2-layer MLP with GELU |
| Precision | bfloat16 |
| Sampling | 1 FPS with time-instruction conditioning |
| Max context | up to ~1M tokens |
Results
Scores are as reported in the paper for the Video-Ma²mba family. The row marked (this) corresponds to this checkpoint.
Video-MME (w/o subtitles)
| Model | Total Size | Short | Medium | Long | Overall |
|---|---|---|---|---|---|
| Video-Ma²mba-0.3B (this) | 0.7B | 37.4 | 35.0 | 26.8 | 33.1 |
| Video-Ma²mba-1.3B | 1.8B | 49.4 | 39.2 | 31.9 | 40.3 |
| Video-Ma²mba-2.7B | 3.1B | 57.6 | 42.7 | 35.4 | 45.2 |
LongVideoBench
| Model | Total Size | Val | Test |
|---|---|---|---|
| Video-Ma²mba-0.3B (this) | 0.7B | 34.0 | 34.2 |
| Video-Ma²mba-1.3B | 1.8B | 38.0 | 39.8 |
| Video-Ma²mba-2.7B | 3.1B | 43.0 | 44.2 |
General Video Understanding
| Model | Total Size | ActivityNet-QA | Video-ChatGPT | MVBench |
|---|---|---|---|---|
| Video-Ma²mba-0.3B (this) | 0.7B | 43.8 | 2.69 | 41.1 |
| Video-Ma²mba-1.3B | 1.8B | 50.0 | 2.76 | 44.4 |
| Video-Ma²mba-2.7B | 3.1B | 51.7 | 3.03 | 48.3 |
Model Variants
| Model | LLM (Mamba-2) | Total |
|---|---|---|
| video-ma2mba-0.3b (this) | 370M | 0.7B |
| video-ma2mba-1.3b | 1.3B | 1.8B |
| video-ma2mba-2.7b | 2.7B | 3.1B |
Usage
Video-Ma²mba uses a custom LlavaMambaForCausalLM architecture. For the inference pipeline, video preprocessing, and evaluation scripts, please refer to the official Video-Ma²mba GitHub repository.
Citation
@article{lee2024videoma2mba,
title={Look Every Frame All at Once: Video-Ma2mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing},
author={Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man},
journal={arXiv preprint arXiv:2411.19460},
year={2024}
}
License
This model is released under the Apache 2.0 License.