[ECCV 2026] MoGe4D: Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation
Yanran Zhang*,1, Ziyi Wang*,1, Wenzhao Zheng†,1, Zheng Zhu2, Jie Zhou1, Jiwen Lu1
1Department of Automation, Tsinghua University 2GigaAI
*Equal Contribution †Corresponding Author
📄 Paper Summary
Generating interactive and dynamic 4D scenes from a single static image is a core challenge. Existing methods decouple geometry from motion — either generate-then-reconstruct (geometric inconsistency) or reconstruct-then-generate (limited, externally-constrained motion) — causing spatiotemporal inconsistency and poor generalization.
MoGe4D (Motion and Geometry-aware image-to-4D synthesis) is a geometry-conditioned framework that models a scene as dense 4D point trajectories. Starting from an initial geometric prior of the input image, it predicts future time-varying trajectories through a diffusion process, tightly coupling geometric modeling with motion generation. This yields 4D scenes with strong temporal coherence, geometry-aware consistency, and compelling novel-view synthesis.
Contributions:
- TrajScene-60K — 60K videos with dense 4D point trajectories (3M+ frames, ~12B 3D points).
- 4D-STraG — a diffusion trajectory generator with depth-guided motion normalization and a Motion Perception Module (MPM).
- 4D-ViSM — a view-synthesis module rendering the 4D representation under arbitrary camera trajectories.
🧱 Model Structure
This repository releases the three trained components of MoGe4D:
| Path | Size | Description |
|---|---|---|
4D-STraG/diffusion_pytorch_model.safetensors | ~31.9 GiB | 4D Scene Trajectory Generator (diffusion model, built on Wan2.1-14B) |
4D-ViSM/lora_diffusion_pytorch_model.safetensors | ~1.36 GiB | 4D View Synthesis Module (LoRA adapter) |
VAE/vae/pytorch_model.bin | ~484 MiB | Motion-sensitive VAE for trajectory signals |
VAE/{encoder,decoder}_prompt/pytorch_model.bin | ~1–2 MiB | VAE prompt encoder/decoder |
VAE/{optimizer.bin, scheduler.bin, random_states_0.pkl} | — | Training states for the VAE (optional, for resuming training) |
🛠️ Usage
1. Set up the environment
git clone https://github.com/Zhangyr2022/MoGe4D.git
cd MoGe4D
conda create -n MoGe4D python=3.10 && conda activate MoGe4D
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
pip install -r requirements.txt
Install third-party deps: UniDepth and diff-gaussian-rasterization.
2. Download the checkpoints
huggingface-cli download Yanran21/MoGe4D --local-dir ./models --resume-download
(Also place the base backbones Wan2.1-Fun-V1.1-14B-Control/InP, OmniMAE, and UniDepth under ./models.)
3. Inference
bash scripts/inference/infer.sh # whole pipeline: image → 4D scene → multi-view videos
See the GitHub README for training scripts and details.
📊 Results
MoGe4D delivers superior geometric consistency, dynamic realism, and visual fidelity over decoupled approaches (e.g., generate-then-reconstruct with VGGT). Please refer to the paper for quantitative metrics and qualitative comparisons.
📖 Citation
@inproceedings{zhang2026moge4d,
title={Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation},
author={Zhang, Yanran and Wang, Ziyi and Zheng, Wenzhao and Zhu, Zheng and Zhou, Jie and Lu, Jiwen},
booktitle={European Conference on Computer Vision (ECCV)},
year={2026}
}
📧 Contact
- Yanran Zhang — zhangyr21@mails.tsinghua.edu.cn