R3D2-BIG
One-step diffusion for realistic lighting and shadows around inserted 3D assets. This is the regular VAE version of R3D2.
Non-commercial use only. This model is released under the Waymo Dataset License Agreement. All downstream use or modification is subject to its terms, including Section 4. Full agreements and attribution: LICENSE.md, NOTICE.
Use with Diffusers
Tested with PyTorch 2.6.0, Diffusers 0.40.0 and Transformers 5.17.0. A CUDA GPU is required for the example below:
pip install torch==2.6.0 diffusers==0.40.0 transformers==5.17.0 peft
import torch
from diffusers import DiffusionPipeline
from PIL import Image
pipe = DiffusionPipeline.from_pretrained(
"bertaveira/R3D2-big",
trust_remote_code=True,
torch_dtype=torch.float16,
).to("cuda")
pipe(Image.open("input.png").convert("RGB")).images[0].save("output.png")
Use with the original repository
Inside zenseact/R3D2, after uv sync:
uv run huggingface-cli download bertaveira/R3D2-big --local-dir checkpoints/R3D2-big
uv run r3d2/scripts/eval.py --model-path checkpoints/R3D2-big \
--image-path input.png --output-path output.png
The repository CLI uses FP32. For lower GPU memory use, use the FP16 example above.
Credits
Built on SD-Turbo and TAESD, using the Waymo Open Dataset. Powered by Stability AI. For citation, see the paper's repository.