MatFuse — Controllable Material Generation with Diffusion Models

MatFuse generates tileable PBR material maps (diffuse, normal, roughness, specular) from text, reference images, sketches, and/or color palettes.

Paper: MatFuse: Controllable Material Generation with Diffusion Models — CVPR 2024 Project page: https://gvecchio.com/matfuse/

Quick Start

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "gvecchio/MatFuse",
    trust_remote_code=True,
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

result = pipe(
    text="red brick wall",
    num_inference_steps=50,
    guidance_scale=4.0,
    generator=torch.Generator("cuda").manual_seed(42),
)

result["diffuse"][0].save("diffuse.png")
result["normal"][0].save("normal.png")
result["roughness"][0].save("roughness.png")
result["specular"][0].save("specular.png")

Conditioning Inputs

All conditions are optional and freely composable:

from PIL import Image

result = pipe(
    image=Image.open("reference.png"),
    text="rough stone texture",
    palette=[(120, 80, 60), (90, 60, 40), (150, 110, 80), (70, 50, 30), (180, 140, 100)],
    num_inference_steps=50,
    guidance_scale=4.0,
)

Architecture

📜 Citation

@inproceedings{vecchio2024matfuse,
  author    = {Vecchio, Giuseppe and Sortino, Renato and Palazzo, Simone and Spampinato, Concetto},
  title     = {MatFuse: Controllable Material Generation with Diffusion Models},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024},
  pages     = {4429-4438}
}

License

This project is licensed under the MIT License.