This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
Pipeline Type: Wan22VaceBlocks
Description: Modular pipeline for controllable video generation using Wan2.2 VACE.
This pipeline uses a 5-block architecture that can be customized and extended.
Example Usage
[TODO]
Pipeline Architecture
This modular pipeline is composed of the following blocks:
- text_encoder (
WanTextEncoderStep)- Text Encoder step that generate text_embeddings to guide the video generation
- vace_encoder (
WanVaceEncoderStep)- Vace Encoder step that preprocesses the control video, mask and reference images and encodes them into the conditioning latents used by the VACE control branch of the transformer
- denoise (
Wan22VaceCoreDenoiseStep)- denoise block that takes encoded text and vace conditioning latents and runs the denoising process.
- trim_latents (
WanVaceTrimReferenceLatentsStep)- Step that removes the prepended reference image frames from the denoised latents before decoding
- decode (
WanVaeDecoderStep)- Step that decodes the denoised latents into images
Model Components
- text_encoder (
UMT5EncoderModel) - tokenizer (
AutoTokenizer) - guider (
ClassifierFreeGuidance) - transformer (
WanVACETransformer3DModel) - vae (
AutoencoderKLWan) - video_processor (
VideoProcessor) - scheduler (
UniPCMultistepScheduler) - guider_2 (
ClassifierFreeGuidance) - transformer_2 (
WanVACETransformer3DModel)
Configuration Parameters
boundary_ratio (default: 0.875): The boundary ratio to divide the denoising loop into high noise and low noise stages.
Input/Output Specification
Inputs:
prompt(None, optional): No description providednegative_prompt(None, optional): No description providedmax_sequence_length(None, optional, defaults to512): No description providedvideo(list, optional): The control video to condition the generation on. If not provided, an empty video is used.mask(list, optional): The mask that defines which video regions to condition on (black) and which to generate (white). Can only be passed ifvideois passed as well.reference_images(Image | list, optional): One or more reference images as extra conditioning for the generation.conditioning_scale(float | list | Tensor, optional, defaults to1.0): The conditioning scale applied in each control layer of the model. If a float, it is applied uniformly to all layers; a list or tensor must have the same length as the number of control layers.height(None, optional): No description providedwidth(None, optional): No description providednum_frames(int, optional, defaults to81): No description providedgenerator(None, optional): No description providednum_videos_per_prompt(None, optional, defaults to1): No description providednum_inference_steps(None, optional, defaults to50): No description providedtimesteps(None, optional): No description providedsigmas(None, optional): No description providedlatents(Tensor | NoneType, optional): No description providedattention_kwargs(None, optional): No description providedoutput_type(str, optional, defaults tonp): The output type of the decoded videos
Outputs:
videos(list): The generated videos.