IRis — hybrid-003 checkpoint

Checkpoint 002_re_015000 (15k iterations) of the IRis hybrid architecture for blind image restoration, from the Master's thesis "Blind Image Restoration via Dual-Conditioned Latent Diffusion" (University of Pisa).

Architecture

• unet/ — Stable Diffusion 2 UNet with the first conv expanded to 8 input channels ([degraded latent (4), noisy latent (4)]), trained end-to-end.
• controlnet/ — ControlNet branch conditioned on the degraded image in pixel space, trained jointly with the UNet.
• scheduler/scheduler_config.json — DDIM scheduler config (used with timestep_spacing="trailing" and rescale_betas_zero_snr=True).
• hybrid_003_config.json — training configuration metadata ({"architecture": "hybrid-003", "arniqa_enabled": false}).

ARNIQA conditioning is disabled for this checkpoint (arniqa_enabled: false), so the empty text embedding is used for cross-attention.

Usage

The custom pipeline classes live in the IRis repository (marigold package). Minimal example:

from diffusers import AutoencoderKL, ControlNetModel, DDIMScheduler, UNet2DConditionModel
from transformers import CLIPTextModel, CLIPTokenizer
from marigold import MarigoldHybridControlNetArniqa003PipelinePatched

ckpt = "ccalzerano72/IRis-hybrid-003"
sd2 = "sd2-community/stable-diffusion-2-1"

pipe = MarigoldHybridControlNetArniqa003PipelinePatched(
    unet=UNet2DConditionModel.from_pretrained(f"{ckpt}/unet", torch_dtype=torch.float16),
    controlnet=ControlNetModel.from_pretrained(f"{ckpt}/controlnet", torch_dtype=torch.float16),
    vae=AutoencoderKL.from_pretrained(sd2, subfolder="vae", torch_dtype=torch.float16),
    scheduler=DDIMScheduler.from_pretrained(sd2, subfolder="scheduler"),
    text_encoder=CLIPTextModel.from_pretrained(sd2, subfolder="text_encoder", torch_dtype=torch.float16),
    tokenizer=CLIPTokenizer.from_pretrained(sd2, subfolder="tokenizer"),
    default_denoising_steps=5,
    default_processing_resolution=768,
    patch_size=768,
    overlap_ratio=0.25,
    blend_mode="gaussian",
).to("cuda")

out = pipe(input_image, denoising_steps=5, processing_res=768)
out.restored_img.save("restored.png")

See script/hybrid_controlnet_restoration/run.py in the IRis repository for the full inference script (including the scheduler timestep_spacing/rescale_betas_zero_snr fix required to match training).

License

Model weights: OpenRAIL++-M (see LICENSE in this repository), including the use restrictions in Attachment A. Copyright (c) 2026 Carmelo Calzerano, University of Pisa.