GaussianFormer-V17b
GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization. It adapts
RenderFormer (SIGGRAPH 2025), a transformer renderer for
triangle meshes: each Gaussian's 14 parameters [pos(3), scale(3), quat(4), rgb(3), opacity(1)] become one scene
token, and the two-stage architecture (view-independent scene encoder, view-dependent ray decoder with a DPT head)
is kept and warm-started from RenderFormer's weights.
This checkpoint (V17, epoch 36 of its 512 px stage) replaces V10b as the recommended model.
Code: github.com/SVLwoof/gaussianformer
Training
- Data: 13,405 single-object scans from Objaverse_Splats, each pruned from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians). Targets are gsplat renders of the full, unpruned splat: 14 orbit views per object at 512 px.
- Stage A: 60 epochs at 256 px, log-HDR L1 loss.
- Stage B: 36 epochs at 512 px, log-HDR L1 + LPIPS-VGG (0.5 each), cosine LR 5e-5 to 5e-7, 8 GPUs.
Results
Held-out set of 1,806 objects not seen in training, 4 views each, PSNR on the object's bounding box against the full-splat ground truth:
| PSNR (dB) | |
|---|---|
| gsplat rasterization of the same 20k-Gaussian input | 44.94 |
| GaussianFormer-V17b | 30.29 |
The model reproduces shape, color and coarse texture well but blurs fine detail (thin strokes, text, small patterns); that gap to rasterization is what current work targets.
How to use
import torch
from gaussianformer.pipelines.rendering_pipeline import GaussianFormerRenderingPipeline
pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v17b")
pipeline.to(torch.device("cuda"))
# gaussians [B, N, 14] (pos | scale | quat wxyz | rgb | opacity), mask [B, N] bool,
# c2w [B, V, 4, 4] (Blender convention), fov [B, V, 1] in degrees
imgs = pipeline(gaussians=g, mask=m, c2w=c2w, fov=fov, resolution=512, torch_dtype=torch.float16)
# imgs: [B, V, 512, 512, 3], linear HDR
From the command line (reads an HDF5 scene, writes EXR + PNG per view):
uv run python infer_gaussian.py --h5_file scene.h5 --model_id shahafvl/gaussianformer-v17b --output_dir out/
Limitations
- Trained on isolated single objects centred in [-0.45, 0.45]^3, cameras at distance ~1.7 with 45° FOV. Scenes far outside that range degrade.
- Inputs of about 20k Gaussians; scene-encoder attention is quadratic in the number of Gaussians.
- Fine detail is noticeably softer than rasterizing the same splat.
License
Trained on Objaverse_Splats, a subset of Objaverse whose terms restrict commercial use; this checkpoint is CC-BY-NC-4.0, research and non-commercial use only.
Citation
GaussianFormer builds on RenderFormer; please cite:
@inproceedings{zeng2025renderformer,
title = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination},
author = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
booktitle = {ACM SIGGRAPH 2025 Conference Papers},
year = {2025}
}