ShapeR
ShapeR introduces a novel approach to metric shape generation. Given an input image sequence, preprocessing extracts per-object metric sparse SLAM points, images, poses, and captions using off-the-shelf methods. A rectified flow transformer operating on VecSet latents conditions on these multimodal inputs to generate a shape code, which is decoded into the object’s mesh. By applying the model object-centrically to each detected object, we obtain a metric reconstruction of the entire scene.
Project Page | Arxiv | Paper | Video | HF-Model | HF Evaluation Dataset
Usage
Clone the repository and follow the INSTALL.md instructions to install the required dependencies.
Inference
python infer_shape.py --input_pkl <sample.pkl> --config balance --output_dir output
Arguments:
--input_pkl: Path to preprocessed pickle file (relative todata/)--config: Preset configurationquality: 16 views, 50 steps (best quality, slowest)balance: 16 views, 25 steps (recommended)speed: 4 views, 10 steps (fastest)
--output_dir: Output directory for meshes and visualizations--do_transform_to_world: Transform output mesh to world coordinates--remove_floating_geometry: Remove disconnected mesh components (default: on)--simplify_mesh: Reduce mesh complexity (default: on)--save_visualization: Save comparison visualization (default: on)
Example
python infer_shape.py --input_pkl gaia_29093018117008733__20250415_125853.pkl --config balance
Output:
<name>.glb: Reconstructed 3D meshVIS__<name>.jpg: Visualization comparing input, prediction, and ground truth
Data Format
This codebase assumes that A sequence has already been processed using the Aria MPS pipeline, along with an 3D object instance detector, resulting in the pickle files with the preprocessed data required for ShapeR ingestion.
We release the ShapeR Evaluation Dataset containing preprocessed samples from Aria glasses captures. Each sample is a pickle file with point clouds, multi-view images, camera parameters, text captions, and ground truth meshes.
For a detailed walkthrough of the data format, see the explore_data.ipynb notebook which includes:
- Complete pickle file structure with all keys and their dimensions
- Interactive 3D visualization of point clouds and meshes
- Camera position visualization
- Image and mask grid displays
- DataLoader usage examples for both SLAM and RGB variants
- Explanation of view selection strategies
Project Structure
ShapeR/
├── infer_shape.py # Main inference script
├── explore_data.ipynb # Data exploration notebook
├── dataset/
│ ├── shaper_dataset.py # Dataset and dataloader
│ ├── image_processor.py # View selection and image preprocessing
│ └── point_cloud.py # Point cloud to SparseTensor
├── model/
│ ├── flow_matching/
│ │ └── shaper_denoiser.py # Flow matching denoiser
│ ├── vae3d/
│ │ └── autoencoder.py # 3D VAE for mesh generation
│ ├── pointcloud_encoder.py # Sparse 3D convolution encoder
│ └── dino_and_ray_feature_extractor.py # Image feature extraction
├── preprocessing/
│ └── helper.py # Fisheye rectification, camera utils
├── postprocessing/
│ └── helper.py # Mesh cleanup, visualization
├── checkpoints/ # Model weights (downloaded automatically)
└── data/ # Input pickle files
License
The ShapeR model are licensed under CC-BY-NC. See the LICENSE file for details. However portions of the project use checkpoints under separate license terms: see NOTICE. e.g. DinoV2 is licensed under Apache-2.0 license.
Citation
If you find ShapeR useful for your research, please cite our paper:
@misc{siddiqui2026shaperrobustconditional3d,
title={ShapeR: Robust Conditional 3D Shape Generation from Casual Captures},
author={Yawar Siddiqui and Duncan Frost and Samir Aroudj and Armen Avetisyan and Henry Howard-Jenkins and Daniel DeTone and Pierre Moulon and Qirui Wu and Zhengqin Li and Julian Straub and Richard Newcombe and Jakob Engel},
year={2026},
eprint={2601.11514},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2601.11514},
}