Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
Latent-Foresight learns a latent world model end to end: an autoencoder compresses frozen DINOv2 features into a compact latent space, and a diffusion transformer (DiT), trained with a flow-matching (rectified flow) objective, predicts the latent of future frames. The autoencoder and the predictor are trained jointly, so the latent space is shaped to be predictable. Off-the-shelf DPT heads then decode the predicted features into semantic segmentation, depth and surface normals.
This repository contains the checkpoints trained on Cityscapes.
Paper
Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
Code
The official implementation can be found on GitHub: https://github.com/Sta8is/Latent-Foresight
Checkpoints
| File | Description |
|---|---|
latent_foresight_cs_lowres.ckpt | Latent-Foresight trained on Cityscapes at low resolution (224x448) |
latent_foresight_cs.ckpt | Latent-Foresight fine-tuned on Cityscapes at high resolution (448x896) |
dinov2_stats.pth | Per-channel mean and std used to normalize the DINOv2 features (--meanstd_ckpt) |
head_segm.pth | DPT head for semantic segmentation |
head_depth.pth | DPT head for depth estimation |
head_normals.pth | DPT head for surface normal estimation |
Sample Usage
Install the environment with uv and download the checkpoints:
git clone https://github.com/Sta8is/Latent-Foresight
cd Latent-Foresight
uv sync
source .venv/bin/activate
hf download Sta8is/Latent-Foresight_cs --local-dir checkpoints
For training and evaluation instructions, refer to the GitHub repository.
Citation
If you found Latent-Foresight useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!
@article{karypidis2026latent,
title={Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models},
author={Karypidis, Efstathios and Gidaris, Spyros and Komodakis, Nikos},
journal={arXiv preprint arXiv:2610.01942},
year={2026}
}
Acknowledgements
Our code is partially based on Dino-Foresight. We also thank the authors of DINOv2 and DPT for their work and open-source code.