Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference

Abstract

Model Description

This model is fine-tuned from stable-diffusion-xl-base-1.0. It has been trained on 4,000 prompts for 10 epochs.

This is a merged checkpoint that combines the LoRA checkpoint with the base model stable-diffusion-xl-base-1.0. If you want to access the LoRA checkpoint, please visit SPO-SDXL_4k-p_10ep_LoRA. We also provide a LoRA checkpoint compatible with stable-diffusion-webui, which can be accessed here.

A quick example

from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel
import torch

# load pipeline
inference_dtype = torch.float16
pipe = StableDiffusionXLPipeline.from_pretrained(
    "SPO-Diffusion-Models/SPO-SDXL_4k-p_10ep", 
    torch_dtype=inference_dtype,
)
vae = AutoencoderKL.from_pretrained(
    'madebyollin/sdxl-vae-fp16-fix',
    torch_dtype=inference_dtype,
)
pipe.vae = vae
pipe.to('cuda')

generator=torch.Generator(device='cuda').manual_seed(42)
image = pipe(
    prompt='a child and a penguin sitting in front of the moon',
    guidance_scale=5.0,
    generator=generator,
    output_type='pil',
).images[0]
image.save('moon.png')

Citation

If you find our work or codebase useful, please consider giving us a star and citing our work.

@article{liang2024step,
  title={Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization},
  author={Liang, Zhanhao and Yuan, Yuhui and Gu, Shuyang and Chen, Bohan and Hang, Tiankai and Cheng, Mingxi and Li, Ji and Zheng, Liang},
  journal={arXiv preprint arXiv:2406.04314},
  year={2024}
}