New verison is available: https://huggingface.co/Disty0/sotediffusion-v2

SoteDiffusion Wuerstchen3

Anime finetune of Würstchen V3.

Release Notes

• This release is sponsored by fal.ai/grants
• Trained on 6M images for 3 epochs using 8x A100 80G GPUs.

API Usage

This model can be used via API with Fal.AI
For more details: https://fal.ai/models/fal-ai/stable-cascade/sote-diffusion

UI Guide

SD.Next

URL: https://github.com/vladmandic/automatic/

Go to Models -> Huggingface and type Disty0/sotediffusion-wuerstchen3-decoder into the model name and press download.
Load Disty0/sotediffusion-wuerstchen3-decoder after the download process is complete.

Prompt:

newest, extremely aesthetic, best quality,

Negative Prompt:

very displeasing, worst quality, monochrome, realistic, oldest, loli,

Parameters:
Sampler: Default

Steps: 30 or 40
Refiner Steps: 10

CFG: 7
Secondary CFG: 2 or 1

Resolution: 1024x1536, 2048x1152
Anything works as long as it's a multiply of 128.

ComfyUI

Please refer to CivitAI: https://civitai.com/models/353284

Code Example

pip install diffusers

import torch
from diffusers import StableCascadeCombinedPipeline

device = "cuda"
dtype = torch.bfloat16 # or torch.float16
model = "Disty0/sotediffusion-wuerstchen3-decoder"

pipe = StableCascadeCombinedPipeline.from_pretrained(model, torch_dtype=dtype)

# send everything to the gpu:
pipe = pipe.to(device, dtype=dtype)
pipe.prior_pipe = pipe.prior_pipe.to(device, dtype=dtype)

# or enable model offload to save vram:
# pipe.enable_model_cpu_offload()



prompt = "newest, extremely aesthetic, best quality, 1girl, solo, cat ears, pink hair, orange eyes, long hair, bare shoulders, looking at viewer, smile, indoors, casual, living room, playing guitar,"
negative_prompt = "very displeasing, worst quality, monochrome, realistic, oldest, loli,"
output = pipe(
    width=1024,
    height=1536,
    prompt=prompt,
    negative_prompt=negative_prompt,
    decoder_guidance_scale=2.0,
    prior_guidance_scale=7.0,
    prior_num_inference_steps=30,
    output_type="pil",
    num_inference_steps=10
).images[0]

## do something with the output image

Training:

Software used: Kohya SD-Scripts with Stable Cascade branch.
https://github.com/kohya-ss/sd-scripts/tree/stable-cascade

GPU used: 8x Nvidia A100 80GB
GPU Hours: 220

Base

Final

Dataset:

GPU used for captioning: 1x Intel ARC A770 16GB
GPU Hours: 350

Model used for captioning: SmilingWolf/wd-swinv2-tagger-v3
Model used for text: llava-hf/llava-1.5-7b-hf

Command:

python /mnt/DataSSD/AI/Apps/kohya_ss/sd-scripts/finetune/tag_images_by_wd14_tagger.py --model_dir "/mnt/DataSSD/AI/models/wd14_tagger_model" --repo_id "SmilingWolf/wd-swinv2-tagger-v3" --recursive --remove_underscore --use_rating_tags --character_tags_first --character_tag_expand --append_tags --onnx --caption_separator ", " --general_threshold 0.35 --character_threshold 0.50 --batch_size 4 --caption_extension ".txt" ./

Note:

• Smallest size is 1280x600 | 768.000 pixels
• Deduped based on image similarity using czkawka-cli
• Around 120K very high quality images got intentionally duplicated 5 times, making the total image count 6.2M

Tags:

Model is trained with random tag order but this is the order in the dataset if you are interested:

aesthetic tags, quality tags, date tags, custom tags, rating tags, character, series, rest of the tags

Date:

Aesthetic Tags:

Model used: shadowlilac/aesthetic-shadow-v2

Quality Tags:

Model used: https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth

Rating Tags:

Custom Tags:

Limitations and Bias

Bias

• This model is intended for anime illustrations.
  Realistic capabilites are not tested at all.

Limitations

• Can fall back to realistic.
  Add "realistic" tag to the negatives when this happens.
• Far shot eyes and hands can be bad.

License

SoteDiffusion models falls under Fair AI Public License 1.0-SD license, which is compatible with Stable Diffusion models’ license. Key points:

1. Modification Sharing: If you modify SoteDiffusion models, you must share both your changes and the original license.
2. Source Code Accessibility: If your modified version is network-accessible, provide a way (like a download link) for others to get the source code. This applies to derived models too.
3. Distribution Terms: Any distribution must be under this license or another with similar rules.
4. Compliance: Non-compliance must be fixed within 30 days to avoid license termination, emphasizing transparency and adherence to open-source values.

Notes: Anything not covered by Fair AI license is inherited from Stability AI Non-Commercial license which is named as LICENSE_INHERIT.