TT-VidT
Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining
NeurIPS 2026 (main track)
TT-VidT is a self-supervised video encoder built for motion. A DINOv3 ViT-B/16 processes every frame independently (the appearance path), while a compact Temporal Transfer pathway turns each frame into a few motion tokens that exchange information across time. This repository holds the pretrained TT3D encoder (195.5M parameters).
Model
- Encoder: 12 DINOv3 ViT-B/16 layers interleaved with 12 Temporal Transfer (TT3D) layers. Each TT3D layer runs block-causal attention over the frame's K = 8 motion tokens together with its 4x-downsampled spatial tokens, and writes the result back to the spatial stream.
- Pretraining objective: Diff Compression. A DiT decoder reconstructs every later frame from the first frame's features plus that frame's motion tokens, so the motion tokens carry what the first frame cannot explain.
| Parameters | 195.5M (incl. the 85.1M DINOv3 spatial path) |
| Input | 8 RGB frames, 256 x 256, pixels in [-1, 1] (mean = std = 0.5), sampled at 6 fps in pretraining |
| Output | motion_output: one 768-d motion embedding per frame, [B, T, 1, 768] |
| Weights | fp32 safetensors, encoder only |
Quick start
With transformers only (the model code ships in this repository):
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained("KBlueLeaf/TTVidT", trust_remote_code=True).cuda().eval()
video = torch.rand(1, 8, 3, 256, 256, device="cuda") * 2 - 1 # [B, T, C, H, W] in [-1, 1]
with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
motion = model(video).motion_output # [B, T, 1, 768]
With the TT-VidT codebase (training, evaluation, feature extraction):
from ttvidt.hub import load_model
model = load_model("KBlueLeaf/TTVidT", device="cuda")
motion = model.encoder(video).motion_output
Loading needs no access to the (gated) DINOv3 base weights: every weight is in this repository.
Results
Frozen attentive probe on the motion embeddings, top-1 accuracy (%), mean of 3 seeds, 8 frames (evaluation protocol of the paper):
| HMDB51 | ARID | IARD | Jester | SSv2 |
|---|---|---|---|---|
| 25.2 | 36.1 | 86.0 | 72.9 | 25.4 |
For the full study (24 architecture–objective pairs at matched scale, fine-tuning and diagnostics), see the paper.
Possible downstream uses
The encoder gives a compact sequence of per-frame motion tokens alongside the DINOv3 appearance features. Some directions we think are worth trying:
- From image models to video models: pair an existing image model with the motion token sequence to get a video model for understanding in the broad sense: classification, retrieval, captioning, question answering, or any other task.
- Generation and motion transfer: use the motion tokens as a conditioning signal for video generation, or take them from one clip and apply them to another subject or scene.
[!TIP] Further exploration and feedback are very welcome, and so are attempts at larger scale (bigger backbones, more data, longer training). Please open an issue or a discussion on GitHub or in the Community tab.
Related resources
| Resource | Link |
|---|---|
| Project page | kohakublueleaf.github.io/TTVidT |
| Paper | huggingface.co/papers/2609.33419 · arXiv:2609.33419 |
| Source code (training, evaluation, all paper configs) | github.com/KohakuBlueleaf/TTVidT |
| Pretrained DiT decoders (Diff Compression and the other objectives) | KBlueLeaf/TTVidT-decoders |
Files
| File | Content |
|---|---|
config.json | architecture and loader configuration (transformers + TT-VidT codebase) |
model.safetensors | fp32 encoder weights |
*.py | encoder code for trust_remote_code (needs only torch and transformers) |
assets/ | figures of this card |
License
Apache-2.0. The spatial path is initialised from DINOv3, which is released under its own license.