TDV: You Don't Need Strong Assumptions — Visual Representation Learning via Temporal Differences
Ninad Daithankar*, Alexi Gladstone*, Yann LeCun, Heng Ji University of Illinois Urbana-Champaign · New York University (*Equal Contribution)
Paper | Project Page | GitHub | HF Paper
Overview
TDV is a self-supervised video representation learning method built on a single causal assumption: the past causes the future. Rather than relying on strong inductive biases like augmentations, masking, or cropping, TDV jointly trains a frame encoder and a motion encoder such that:
current frame representation + encoded motion = next frame representation
The motion encoder processes RGB differences between consecutive frames, conditioned on the current frame representation via cross-attention. Predictions are supervised using MSE loss against an EMA teacher encoder, with DINO-style categorical cross-entropy to prevent representation collapse.
TDV matches DINO and iBOT on dense segmentation tasks while surpassing both on optical flow and stereo depth — tasks that directly reward temporally structured representations.
Checkpoints
All checkpoints are in the checkpoints/ folder. Optimizer states and training metadata have been stripped — these are inference-ready weights.
All models are pretrained on Something-Something v2 (SSv2).
| File | Model | Size | Notes |
|---|---|---|---|
checkpoints/tdv-base.pth | TDV ViT-Base | ~1.2 GB | TDV frame encoder (our method) |
checkpoints/tdv-small.pth | TDV ViT-Small | ~400 MB | TDV frame encoder (our method) |
checkpoints/dino-base.pth | DINO ViT-Base | ~739 MB | DINO baseline (teacher weights) |
checkpoints/dino-small.pth | DINO ViT-Small | ~220 MB | DINO baseline (teacher weights) |
checkpoints/ibot-base.pth | iBOT ViT-Base | ~762 MB | iBOT baseline (teacher weights) |
checkpoints/ibot-small.pth | iBOT ViT-Small | ~243 MB | iBOT baseline (teacher weights) |
DINO and iBOT checkpoints are our reproductions trained with online KNN monitoring, used as baselines in the paper.
Usage
Loading TDV
TDV checkpoints are PyTorch Lightning checkpoints with the optimizer states stripped. Load the state_dict directly:
import torch
ckpt = torch.load("checkpoints/tdv-base.ckpt", map_location="cpu")
state_dict = ckpt["state_dict"]
# The frame encoder keys are prefixed with "model.frame_encoder."
# Strip the prefix to load into a standalone ViT:
encoder_state_dict = {
k.replace("model.frame_encoder.", ""): v
for k, v in state_dict.items()
if k.startswith("model.frame_encoder.")
}
For the full architecture and eval setup see github.com/ninaddaithankar/tdv.
Loading DINO / iBOT baselines
import torch
ckpt = torch.load("checkpoints/dino-base.pth", map_location="cpu")
teacher_weights = ckpt["teacher"] # final model weights
# student weights also available: ckpt["student"]
Results
TDV is evaluated on dense spatial tasks where temporal structure matters most:
- Optical Flow (MPI-Sintel): Lower EPE than DINO and iBOT baselines
- Stereo Depth (SceneFlow): Lower bad-pixel rates at 0.5px and 1px thresholds
- Semantic Segmentation (ADE20K / Cityscapes): Matches DINO and iBOT baselines
See the paper for full tables and ablations.
Citation
@misc{daithankar2026dontneedstrongassumptions,
title={You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences},
author={Ninad Daithankar and Alexi Gladstone and Yann LeCun and Heng Ji},
year={2026},
eprint={2606.15956},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.15956},
}