NSFW enabled capable model that still has base model capability and functions. Betas are all testing integrations and bases and are just samples ahead of inegrating sulphur h3 which will be the main data backbone and v1.0.
beta_6 is a built mostly-reference hybrid with a custom turbo meant to mimic how Fastv2 looks and sounds. Samples on 10/3 shift, fully dense or with Fastv2 VSA sparse node settings on multiple samplers like er_sde, euler, LCM, res_multistep, all having some different behaviors with prompt. It's a slow 8-12 step reference model in that configuration. 4step turbo loras can be run on top of it to reduce that. SLA sparsity seems to destroy it, avoid using that on any workflows. It's ideal configuration is with full gates and Fastv2 turbo and VSA gates when they have a reference solution figured out. The attempt to do that is the unstable version which has multiple issues but looks and sounds perfect at 8 steps.
beta_5 is built with a different normalization technique. It's built off 7 different concept-grouped grafts and consensus merges from over 20 loras, no direct lora merges. Lora credits are on the civit page. It allows for a non-turbo version which is avaliable. The files with TURBO have a hybrid turbo delta fusion baked in, saving 4.2 gigabytes of memory instead of having to load both ref/fl turbos. Other concept loras will also load on top more readily, usually needing lower strength 0.2-0.6.
Sampling and prompt are the key to quality and success. Non-turbo full step audio will always be better than audio on turbo versions.
Some working sampling setups:
er_sde/beta57 4-6 steps
res_multistep/simple 6-8 steps (my go-to for best motion quality)
LCM/simple 6-8 steps (best turbo audio)
LCM/beta 4-6 steps
Euler/simple 4-8 steps
Int8 versions https://huggingface.co/cicalooo/10Eros-Max-h3-int8-convrot
beta_4 and beta_3 are corrupted test versions. beta_5 is the only true functional one so far. No cache or spectrum if doing reference, they cause accuracy loss. Hybrid model is built on the delta1024 h3 merge.
Regarding the grafts:
Due to a ton of confusion I've let Claude compile a full MD on the grafting, including code and methodology as it was mostly used to create code, document, and manage the graft project while I tested, architected, and mixed. I was keeping that to myself since maybe it'd actually be nice to have something to myself but after a handful of ignorant comments I'm open sourcing everything from it except the scripts themselves. You can hand the md back to claude or any competent agent and graft in the same way or have them explain it until you understand. No more dumbassery allowed now.
This is an evolving project subject to future fixes in H3 training. Training is clearly problematic, and my branch will rely on Sulphur H3 tunes. But I didn't want to wait for all that so I took the data from older models, grafting it in where the model needs it, grafting it to attn layers at a low level that doesn't disturb H3's visual or audio output quality. That's essentialy it.
The standard H3 community license applies. Because this release now carries transferred character from LTX 2.3, Wan 2.2, and Krea 2, the community licenses for those source models apply as well to the portions of character that came from each.
10Eros Max H3 Transformer (Diffusers Format)
MiniMax-H3 transformer, converted from the 10Eros Max checkpoint
Loading
You MUST use AutoModel.from_pretrained with trust_remote_code=True:
from diffusers import AutoModel
transformer = AutoModel.from_pretrained(
"ibyteohdear/10Eros-Max-Transformer_8stepTURBO_beta6_VBVR_FUSED",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
)
Do NOT use `MiniMaxH3Transformer3DModel.from_pretrained` - it will fail because
this is a pruned checkpoint with compressed AdaLN (rank-8).
# this code need the provided differs from this repo, in root of app
import os
import sys
import tempfile
import re
import random
BASE = tempfile.gettempdir()
IS_HF_SPACES = bool(os.getenv("SPACES_ZERO_GPU"))
HF_TOKEN = os.getenv("HF_TOKEN")
if IS_HF_SPACES:
import spaces
clip_dir=BASE
else:
os.environ["HF_HOME"] = "/mnt/data/huggingface"
os.environ["HF_HUB_DISABLE_XET"] = "1"
os.environ["HF_HUB_DOWNLOAD_TIMEOUT"] = "120"
clip_dir="/mnt/data/huggingface"
import gradio as gr
import torch
import numpy as np
from PIL import Image
from huggingface_hub import snapshot_download, hf_hub_download
diffusers_path = snapshot_download(
repo_id="ibyteohdear/10Eros-Max-Transformer_8stepTURBO_beta6_VBVR_FUSED",
token=HF_TOKEN,
allow_patterns=["diffusers/*"],
)
sys.path.insert(0, diffusers_path)
from diffusers import (
TorchAoConfig,
AutoencoderKLMiniMaxH3,
AutoencoderKLMiniMaxH3Audio,
)
from diffusers.utils.export_utils import encode_video
from torchao.quantization import Int8WeightOnlyConfig
from transformers import Qwen3VLForConditionalGeneration, AutoTokenizer, Qwen3VLProcessor
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from diffusers.modular_pipelines.minimax_h3.before_denoise import (
MiniMaxH3PrepareLatentsStep,
MiniMaxH3PrepareLayoutStep,
MiniMaxH3SetTimestepsStep,
)
from diffusers.modular_pipelines.minimax_h3.before_encoder import (
MiniMaxH3SetupStep,
)
from diffusers.modular_pipelines.minimax_h3.denoise import (
MiniMaxH3DenoiseStep,
)
from diffusers.modular_pipelines.minimax_h3.encoders import (
MiniMaxH3TextEncoderStep,
)
from diffusers.modular_pipelines.minimax_h3.modular_blocks_minimax_h3 import (
MiniMaxH3AutoKeyframeVaeEncoderStep,
MiniMaxH3DecodeStep,
_generation_outputs,
)
from diffusers.modular_pipelines.modular_pipeline import SequentialPipelineBlocks
from diffusers.modular_pipelines.modular_pipeline_utils import OutputParam
import torch.nn.functional as F
from transformers import TorchAoConfig as TransformersTorchAoConfig
from diffusers import AutoModel
# -----------------------------------------------------------------------------
# 1. Zero-GPU decorator
# -----------------------------------------------------------------------------
def zero_gpu_decorator(duration=1500):
def decorator(func):
if IS_HF_SPACES:
return spaces.GPU(duration=duration)(func)
return func
return decorator
#--------------------------------------------------------------------------
# 2. ClipProj
# -----------------------------------------------------------------------------
PROJ_FILENAME = "mmh3-4b-ClipProj-v3.1.safetensors"
MLP_FILENAME = "mmh3-4b-ClipProj-v3.1-mlp.safetensors"
CLIPPROJ_REPO = "NicoLab28/ClipProj-MiniMax-H3"
clipproj_dir = os.path.join(clip_dir, "clipproj")
os.makedirs(clipproj_dir, exist_ok=True)
proj_path = hf_hub_download(
repo_id=CLIPPROJ_REPO,
filename=PROJ_FILENAME,
local_dir=clipproj_dir,
)
mlp_path = hf_hub_download(
repo_id=CLIPPROJ_REPO,
filename=MLP_FILENAME,
local_dir=clipproj_dir,
)
proj_weights = load_file(proj_path)
mlp_weights = load_file(mlp_path)
W = proj_weights["W"].to(torch.bfloat16)
mean_in = proj_weights["mean_in"].to(torch.bfloat16)
std_in = proj_weights["std_in"].to(torch.bfloat16)
mean_out = proj_weights["mean_out"].to(torch.bfloat16)
std_out = proj_weights["std_out"].to(torch.bfloat16)
sink_out = proj_weights.get("sink_out")
if sink_out is not None:
sink_out = sink_out.to(torch.bfloat16)
# -----------------------------------------------------------------------------
# 3. Classes
# -----------------------------------------------------------------------------
class ClipProjModelWrapper(torch.nn.Module):
def __init__(
self,
original_model,
W,
mean_in,
std_in,
mean_out,
std_out,
mlp_weights,
sink_out=None,
):
super().__init__()
self.original_model = original_model
self.register_buffer("W", W)
self.register_buffer("mean_in", mean_in)
self.register_buffer("std_in", std_in)
self.register_buffer("mean_out", mean_out)
self.register_buffer("std_out", std_out)
if sink_out is not None:
self.register_buffer("sink_out", sink_out)
else:
self.sink_out = None
d_in = W.shape[0]
out_dim = W.shape[1]
mlp_in = mlp_weights["mlp.0.weight"].shape[1]
hidden_dim = mlp_weights["mlp.0.weight"].shape[0]
mlp_out = mlp_weights["mlp.2.weight"].shape[0]
self.fc1 = torch.nn.Linear(
d_in,
hidden_dim,
bias=True,
dtype=torch.float16,
)
self.fc2 = torch.nn.Linear(
hidden_dim,
out_dim,
bias=True,
dtype=torch.float16,
)
with torch.no_grad():
self.fc1.weight.copy_(
mlp_weights["mlp.0.weight"]
)
self.fc1.bias.copy_(
mlp_weights["mlp.0.bias"]
)
self.fc2.weight.copy_(
mlp_weights["mlp.2.weight"]
)
self.fc2.bias.copy_(
mlp_weights["mlp.2.bias"]
)
self.fc1.requires_grad_(False)
self.fc2.requires_grad_(False)
self.fc1.eval()
self.fc2.eval()
def forward(self, *args, **kwargs):
kwargs["output_hidden_states"] = True
outputs = self.original_model(
*args,
**kwargs,
)
h = outputs.hidden_states[24]
h_norm = (
h - self.mean_in
) / self.std_in
projected = h_norm @ self.W
with torch.no_grad():
mlp_input = h_norm.to(self.fc1.weight.dtype)
residual = self.fc2(
F.gelu(
self.fc1(mlp_input)
)
).to(projected.dtype)
projected = projected + residual
projected = (
projected * self.std_out
+ self.mean_out
)
if self.sink_out is not None:
projected = projected.clone()
projected[:, 0, :] = self.sink_out.to(
device=projected.device,
dtype=projected.dtype,
)
hidden_states = list(outputs.hidden_states)
while len(hidden_states) <= 50:
hidden_states.append(projected)
hidden_states[50] = projected
outputs.hidden_states = tuple(hidden_states)
outputs.last_hidden_state = projected
return outputs
class MiniMaxH3ConditionerBlocks(SequentialPipelineBlocks):
model_name = "minimax-h3"
block_classes = [
MiniMaxH3SetupStep,
MiniMaxH3TextEncoderStep,
]
block_names = [
"setup",
"text_encoder",
]
@property
def outputs(self):
return [
OutputParam.template("prompt_embeds"),
OutputParam(
"text_token_tags",
description="The per-row modality tag of every row of prompt_embeds.",
),
OutputParam(
"height",
type_hint=int,
description="Resolved height.",
),
OutputParam(
"width",
type_hint=int,
description="Resolved width.",
),
OutputParam(
"num_frames",
type_hint=int,
description="Resolved number of frames.",
),
]
class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
model_name = "minimax-h3"
block_classes = [
MiniMaxH3SetupStep,
MiniMaxH3AutoKeyframeVaeEncoderStep,
MiniMaxH3PrepareLayoutStep,
MiniMaxH3PrepareLatentsStep,
MiniMaxH3SetTimestepsStep,
MiniMaxH3DenoiseStep,
MiniMaxH3DecodeStep,
]
block_names = [
"setup",
"vae_encoder",
"prepare_layout",
"prepare_latents",
"set_timesteps",
"denoise",
"decode",
]
@property
def outputs(self):
return _generation_outputs()
# -----------------------------------------------------------------------------
# 4. Load Qwen3-VL-4B
# -----------------------------------------------------------------------------
text_encoder_4b = Qwen3VLForConditionalGeneration.from_pretrained(
"DreamFast/Qwen3-VL-4b-Heretic",
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
quantization_config=TransformersTorchAoConfig(
Int8WeightOnlyConfig(),
modules_to_not_convert=[
"model.visual",
"model.language_model.embed_tokens",
"model.language_model.norm",
"lm_head"
],
),
)
# -----------------------------------------------------------------------------
# 5. Wrap the model with ClipProj
# -----------------------------------------------------------------------------
original_model = text_encoder_4b.model
model_wrapper = ClipProjModelWrapper(
original_model=original_model,
W=W,
mean_in=mean_in,
std_in=std_in,
mean_out=mean_out,
std_out=std_out,
mlp_weights=mlp_weights,
sink_out=sink_out,
)
original_unwrapped_model = text_encoder_4b.model
text_encoder_4b.model = model_wrapper
text_encoder_4b.config.text_config.num_hidden_layers = 64
text_encoder_4b.requires_grad_(False)
text_encoder_4b.eval()
# -----------------------------------------------------------------------------
# 5.5 Prompt Enhancer (Vision-Aware)
# -----------------------------------------------------------------------------
ENHANCE_SYSTEM_PROMPT = """You are an expert video prompt engineer for MiniMax-H3, a state-of-the-art text-to-video AI model.
Your task: Expand short/rough user prompts into rich, detailed, cinematic descriptions optimized for video generation.
Rules:
- Describe visual details: lighting, camera movement, atmosphere, textures, colors
- Describe motion: how subjects move, camera pans/zooms, environmental dynamics
- Keep it flowing and temporal (beginning → middle → end)
- Avoid negations ("no", "without", "don't") - frame positively
- Aim for 40-80 words
- Output ONLY the enhanced prompt, no explanations, no quotes
Examples:
Input: "A cat sleeping"
Output: A fluffy orange cat curled up on a velvet windowsill, soft afternoon sunlight streaming through sheer curtains, dust particles drifting lazily in golden rays, the cat's chest gently rising and falling with peaceful breaths, occasional ear twitches as distant birds chirp outside.
Input: "They all drink a beer and cheer!"
Output: """
def enhance_prompt(raw_prompt: str, e_prompt_temp: float = 0.7, first_frame: Image.Image = None) -> str:
if not raw_prompt:
return raw_prompt.strip()
if first_frame is not None:
if isinstance(first_frame, np.ndarray):
img_array = first_frame
else:
img_array = np.array(first_frame.convert("RGB"))
content = [
{"type": "image", "image": img_array},
{"type": "text", "text": f"Based on this first frame image, enhance this video prompt: {raw_prompt}"}
]
else:
content = raw_prompt
messages = [
{"role": "system", "content": ENHANCE_SYSTEM_PROMPT},
{"role": "user", "content": content}
]
text = h3_processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
if first_frame is not None:
inputs = h3_processor(
text=[text],
images=[img_array],
return_tensors="pt",
padding=True
)
else:
inputs = h3_processor(
text=[text],
return_tensors="pt",
padding=True
)
inputs = {k: v.to("cuda") if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
text_encoder_4b.model = original_unwrapped_model
text_encoder_4b.to("cuda")
try:
with torch.inference_mode():
outputs = text_encoder_4b.generate(
**inputs,
max_new_tokens=150,
do_sample=True,
temperature=e_prompt_temp,
top_p=0.9,
pad_token_id=h3_processor.tokenizer.pad_token_id,
eos_token_id=h3_processor.tokenizer.eos_token_id,
)
input_len = inputs["input_ids"].shape[1]
generated_ids = outputs[:, input_len:]
enhanced = h3_processor.batch_decode(
generated_ids,
skip_special_tokens=True
)[0].strip()
enhanced = re.sub(r'^["\']|["\']$', '', enhanced)
enhanced = re.sub(r'\n+', ' ', enhanced).strip()
if not enhanced or len(enhanced) < 10:
return raw_prompt.strip()
return enhanced
finally:
text_encoder_4b.model = model_wrapper
# -----------------------------------------------------------------------------
# 6. Build pipeline components
# -----------------------------------------------------------------------------
transformer = AutoModel.from_pretrained(
"ibyteohdear/10Eros-Max-Transformer_8stepTURBO_beta6_VBVR_FUSED",
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
trust_remote_code=True,
quantization_config=TorchAoConfig(
Int8WeightOnlyConfig(),
modules_to_not_convert=[
"proj_in", "audio_proj_in", "context_embedder",
"token_refiner", "norm_out", "proj_out", "audio_proj_out",
],
),
)
transformer.enable_group_offload(
onload_device=torch.device("cuda"),
offload_device=torch.device("cpu"),
offload_type="block_level",
num_blocks_per_group=1,
use_stream=False,
)
vae = AutoencoderKLMiniMaxH3.from_pretrained(
"MiniMaxAI/MiniMax-H3",
subfolder="vae",
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
)
audio_vae = AutoencoderKLMiniMaxH3Audio.from_pretrained(
"MiniMaxAI/MiniMax-H3",
subfolder="audio_vae",
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
)
h3_tokenizer = AutoTokenizer.from_pretrained(
"MiniMaxAI/MiniMax-H3",
subfolder="text_encoder",
low_cpu_mem_usage=True,
)
h3_processor = Qwen3VLProcessor.from_pretrained(
"MiniMaxAI/MiniMax-H3",
subfolder="text_encoder",
low_cpu_mem_usage=True,
)
conditioner_blocks = MiniMaxH3ConditionerBlocks()
conditioner = conditioner_blocks.init_pipeline(
"MiniMaxAI/MiniMax-H3"
)
conditioner.update_components(
text_encoder=text_encoder_4b,
tokenizer=h3_tokenizer,
processor=h3_processor,
)
generator_blocks = MiniMaxH3GeneratorBlocks()
generator = generator_blocks.init_pipeline(
"MiniMaxAI/MiniMax-H3"
)
generator.update_components(
transformer=transformer,
vae=vae,
audio_vae=audio_vae,
)
generator.load_components(dtype=torch.bfloat16)
# -----------------------------------------------------------------------------
# 7. Helpers
# -----------------------------------------------------------------------------
FPS = 24
FRAMES_PER_CHUNK = 17
LATENTS_PER_CHUNK = 5
RESOLUTION_PROFILES = {
"ultra": { "16:9": (1472, 832), "9:16": (832, 1472), "1:1": (1024, 1024) },
"high": { "16:9": (1408, 768), "9:16": (768, 1408), "1:1": (960, 960) },
"medium": { "16:9": (1344, 704), "9:16": (704, 1344), "1:1": (896, 896) },
"low": { "16:9": (1280, 640), "9:16": (640, 1280), "1:1": (832, 832) },
}
MAX_SEED = np.iinfo(np.int32).max
def snap_frames(seconds: float) -> int:
frames = max(1, round(seconds * FPS))
while frames % FRAMES_PER_CHUNK != LATENTS_PER_CHUNK:
frames += 1
return frames
def auto_crop_image(image_path: str, target_width: int, target_height: int, session_id: str = "default") -> str:
print(f"[Auto-Crop] Resizing & Cropping to match target resolution: {target_width}x{target_height}")
img = Image.open(image_path).convert("RGB")
orig_w, orig_h = img.size
target_aspect = target_width / target_height
orig_aspect = orig_w / orig_h
if abs(orig_aspect - target_aspect) < 0.01:
print("[Auto-Crop] Image aspect ratio matches settings. Skipping modifications.")
return image_path
if orig_aspect > target_aspect:
new_h = orig_h
new_w = int(orig_h * target_aspect)
left = (orig_w - new_w) // 2
top = 0
right = left + new_w
bottom = orig_h
else:
new_w = orig_w
new_h = int(orig_w / target_aspect)
left = 0
top = (orig_h - new_h) // 2
right = orig_w
bottom = top + new_h
img_cropped = img.crop((left, top, right, bottom))
print(f"[Auto-Crop] Centered crop applied from {orig_w}x{orig_h} to bounds: left={left}, top={top}, right={right}, bottom={bottom}")
cropped_out_path = os.path.join(BASE, f"cropped_{session_id}.png")
if os.path.exists(cropped_out_path):
try:
os.remove(cropped_out_path)
except Exception:
pass
img_cropped.save(cropped_out_path)
return cropped_out_path
# -----------------------------------------------------------------------------
# 8. Generation
# -----------------------------------------------------------------------------
@zero_gpu_decorator(duration=1500)
def generate_video(prompt, e_prompt_temp, image_pil, end_image_pil, quality, ratio, duration, seed, randomize_seed, use_enhancer, progress=gr.Progress(track_tqdm=True)):
if image_pil is None:
raise ValueError("A starting first frame image is required.")
final_prompt = prompt
if use_enhancer and prompt:
final_prompt = enhance_prompt(prompt, e_prompt_temp, first_frame=image_pil)
width, height = RESOLUTION_PROFILES[quality][ratio]
num_frames = snap_frames(duration)
current_seed = random.randint(0, MAX_SEED) if randomize_seed else int(seed)
session_id = str(current_seed)
if isinstance(image_pil, Image.Image):
temp_first_path = os.path.join(BASE, f"temp_first_{session_id}.png")
image_pil.save(temp_first_path)
cropped_first_path = auto_crop_image(temp_first_path, width, height, session_id=f"first_{session_id}")
image_pil = Image.open(cropped_first_path).convert("RGB")
end_image = None
if end_image_pil is not None and isinstance(end_image_pil, Image.Image):
temp_end_path = os.path.join(BASE, f"temp_end_{session_id}.png")
end_image_pil.save(temp_end_path)
cropped_end_path = auto_crop_image(temp_end_path, width, height, session_id=f"end_{session_id}")
end_image = Image.open(cropped_end_path).convert("RGB")
text_encoder_4b.to("cuda")
with torch.inference_mode():
cond_result = conditioner(
prompt=final_prompt,
image=image_pil,
last_image=end_image,
height=height,
width=width,
num_frames=num_frames,
)
prompt_embeds = cond_result.prompt_embeds
text_token_tags = cond_result.text_token_tags
text_encoder_4b.to("cpu")
del cond_result
torch.cuda.empty_cache()
torch.cuda.ipc_collect()
vae.to("cuda")
audio_vae.to("cuda")
gen_kwargs = {
"prompt_embeds": prompt_embeds,
"text_token_tags": text_token_tags,
"image": image_pil,
"height": height,
"width": width,
"num_frames": num_frames,
"generator": torch.Generator().manual_seed(int(current_seed)),
"output": ["videos", "audio", "sampling_rate"],
"num_inference_steps": 8 + 1,
}
if end_image is not None:
gen_kwargs["last_image"] = end_image
gen_result = generator(**gen_kwargs)
out_path = os.path.join(BASE, f"minimax_h3_gradio_output_{session_id}.mp4")
with torch.inference_mode():
encode_video(
gen_result["videos"][0],
FPS,
out_path,
gen_result["audio"][0],
gen_result["sampling_rate"]
)
return out_path, final_prompt
# -----------------------------------------------------------------------------
# 8.5 Standalone prompt enhancement
# -----------------------------------------------------------------------------
@zero_gpu_decorator(duration=120)
def enhance_prompt_standalone(raw_prompt, e_prompt_temp, image_pil):
if not raw_prompt:
return raw_prompt
enhanced = enhance_prompt(raw_prompt, e_prompt_temp, first_frame=image_pil)
return enhanced
# -----------------------------------------------------------------------------
# 9. UI
# -----------------------------------------------------------------------------
with gr.Blocks(title="MiniMax-H3 FL2VA") as demo:
gr.Markdown("# MiniMax-H3 FL2VA Generator")
gr.Markdown("Using **Qwen3-VL-4B + ClipProj v3.1 + 8 step Lightning**")
with gr.Row():
with gr.Column():
with gr.Row():
prompt_input = gr.Textbox(
label="Prompt",
value="They all drink a beer and cheer!",
lines=2,
)
with gr.Row():
e_prompt_temp = gr.Slider(minimum=0.1, maximum=1.0, value=0.7, step=0.1, label="Enhance prompt temperature")
with gr.Row():
enhance_btn = gr.Button("✨ Enhance Prompt", variant="secondary")
with gr.Row():
image_input = gr.Image(
label="First Frame Image (Required)",
type="pil",
)
end_image_input = gr.Image(
label="Last Frame Image (Optional)",
type="pil",
)
with gr.Row():
quality_tier = gr.Dropdown(
choices=["ultra", "high", "medium", "low"],
value="high",
label="Quality / Canvas Tier"
)
aspect_ratio = gr.Dropdown(
choices=["16:9", "9:16", "1:1"],
value="16:9",
label="Aspect Ratio"
)
with gr.Row():
duration_slider = gr.Slider(
minimum=5.0,
maximum=10.0,
step=0.5,
value=5.0,
label="Duration (seconds)",
)
with gr.Row():
seed_input = gr.Number(
label="Seed",
value=42,
precision=0,
)
with gr.Row():
randomize_seed = gr.Checkbox(value=True, label="Randomize Seed")
with gr.Row():
enhance_toggle = gr.Checkbox(
label="Auto-Enhance on Generate",
value=False,
info="Automatically enhance prompt when clicking Generate"
)
with gr.Row():
generate_btn = gr.Button("Generate", variant="primary")
with gr.Column():
with gr.Row():
video_output = gr.Video(label="Generated Video")
gr.Examples(
examples=[
"They all drink a beer and cheer!",
],
inputs=[prompt_input],
example_labels=[
"They all drink a beer and cheer!",
],
label="💡 Try these:"
)
enhance_btn.click(
fn=enhance_prompt_standalone,
inputs=[prompt_input, e_prompt_temp, image_input],
outputs=[prompt_input],
)
generate_btn.click(
fn=generate_video,
inputs=[
prompt_input,
e_prompt_temp,
image_input,
end_image_input,
quality_tier,
aspect_ratio,
duration_slider,
seed_input,
randomize_seed,
enhance_toggle
],
outputs=[video_output, prompt_input],
)
if __name__ == "__main__":
if IS_HF_SPACES:
demo.queue().launch(debug=False, share=False)
else:
demo.queue().launch(debug=True, share=True)
from huggingface_hub import HfApi, create_repo, hf_hub_download, snapshot_download
from diffusers.models.embeddings import Timesteps, TimestepEmbedding
from diffusers import AutoModel
from safetensors.torch import load_file, save_file
import torch
import torch.nn.functional as F
import os
import json
import shutil
HF_TOKEN = os.environ["HF_TOKEN"]
LOCAL_DIR = "./10eros_h3_transformer_pruned"
REPO_ID = "ibyteohdear/10Eros-Max-Transformer_8stepTURBO_beta6"
SOURCE_FILE = "10Eros_Max_h3_8stepTURBO_beta6_bf16.safetensors" # the un-folded 66.2 GB release
# --- MiniMax-H3 geometry (released config / modeling file) ---
HIDDEN = 5376
NUM_HEADS = 56
HEAD_DIM = 128
INNER_DIM = NUM_HEADS * HEAD_DIM # 7168 (q/k/v rows in a fused qkv_proj)
MODALITY_NUM = 3 # MINIMAX_H3_MODALITY_NUM: video / text / audio
FFN_DIM = 14336 # SwiGLU ffn: fc1 holds value|gate = 2 * 14336 rows
NUM_LAYERS = 50
ADALN_RANK = 8
TIME_EMBED_DIM = 2688
TIME_EMBED_HIDDEN_DIM = 5376
FREQ_DIM = 256
TIME_TABLE_SIZE = 1025
BLOCK_ADALN_OUT = 6 * HIDDEN * MODALITY_NUM # 96768
NORM_OUT_ADALN_OUT = 2 * HIDDEN # 10752
# ---------------------------------------------------------------------------
# Download + load (reuses the HF cache if the 66 GB file is already there)
# ---------------------------------------------------------------------------
checkpoint_path = hf_hub_download(repo_id="TenStrip/10Eros-Max", filename=SOURCE_FILE, token=HF_TOKEN)
print("Loading checkpoint...")
src = load_file(checkpoint_path, device="cpu")
print(f"Source keys: {len(src)}")
print(f"blocks.0.* : {sorted(k for k in src if k.startswith('blocks.0.'))}")
print(f"time_embedder.* : {sorted(k for k in src if k.startswith('time_embedder'))}")
print(f"adaln-ish : {sorted(k for k in src if 'adaln' in k or 'basis' in k or 'mean' in k)}")
# ---------------------------------------------------------------------------
# 1. Rebuild the released timestep path from THIS checkpoint's own weights and
# fit the rank-8 affine subspace the pruned format stores.
#
# Released AdaLN input: x(t) = silu(time_embedder(time_proj(t))), t in [0, 1]
# Pruned format stores: x(t) ~= adaln_mean + c(t) @ adaln_basis
# ---------------------------------------------------------------------------
print("\nFitting the AdaLN affine subspace...")
# The source names the timestep MLP `time_embedder.proj_in` / `.proj_out` (native H3 naming),
# NOT diffusers' `linear_1` / `linear_2`. Identify the two layers by SHAPE, not by name.
te = {k[len("time_embedder."):]: v for k, v in src.items() if k.startswith("time_embedder.")}
if not te:
raise ValueError(f"No `time_embedder.*` keys in the source. Keys present: {sorted(src)[:20]} ...")
print(" source time_embedder keys:", {k: tuple(v.shape) for k, v in sorted(te.items())})
w_first = [k for k, v in te.items() if k.endswith(".weight") and tuple(v.shape) == (TIME_EMBED_HIDDEN_DIM, FREQ_DIM)]
w_second = [k for k, v in te.items() if k.endswith(".weight") and tuple(v.shape) == (TIME_EMBED_DIM, TIME_EMBED_HIDDEN_DIM)]
if len(w_first) != 1 or len(w_second) != 1:
raise ValueError(
f"Could not identify the timestep MLP by shape: first-layer candidates={w_first}, "
f"second-layer candidates={w_second}. Keys seen: {sorted(te)}"
)
s_first = w_first[0][: -len(".weight")]
s_second = w_second[0][: -len(".weight")]
print(f" mapping {s_first} -> linear_1 (in->hidden), {s_second} -> linear_2 (hidden->out)")
for stem, out_dim in ((s_first, TIME_EMBED_HIDDEN_DIM), (s_second, TIME_EMBED_DIM)):
if f"{stem}.bias" not in te:
print(f" WARNING: {stem}.bias missing from the source; assuming zero bias.")
time_proj = Timesteps(num_channels=FREQ_DIM, flip_sin_to_cos=True, downscale_freq_shift=0)
time_embedder = TimestepEmbedding(
in_channels=FREQ_DIM, time_embed_dim=TIME_EMBED_HIDDEN_DIM, out_dim=TIME_EMBED_DIM
)
missing, unexpected = time_embedder.load_state_dict(
{
"linear_1.weight": te[f"{s_first}.weight"].float(),
"linear_1.bias": te.get(f"{s_first}.bias", torch.zeros(TIME_EMBED_HIDDEN_DIM)).float(),
"linear_2.weight": te[f"{s_second}.weight"].float(),
"linear_2.bias": te.get(f"{s_second}.bias", torch.zeros(TIME_EMBED_DIM)).float(),
},
strict=False,
)
if missing or unexpected:
raise ValueError(f"timestep MLP did not line up: missing={missing} unexpected={unexpected}")
time_embedder.eval()
# The table is sampled on a uniform grid of TIME_TABLE_SIZE timesteps in [0, 1], so fit on that grid.
t_grid = torch.linspace(0.0, 1.0, TIME_TABLE_SIZE, dtype=torch.float32)
with torch.no_grad():
x = F.silu(time_embedder(time_proj(t_grid))) # (1025, 2688) float32
x64 = x.to(torch.float64)
# Affine subspace: mean + rank-8 linear span
adaln_mean64 = x64.mean(dim=0) # (2688,)
xc = x64 - adaln_mean64
_, sv, vh = torch.linalg.svd(xc, full_matrices=False)
basis64 = vh[:ADALN_RANK].contiguous() # (8, 2688)
c = xc @ basis64.t() # (1025, 8) = adaln_t_table
recon = adaln_mean64 + c @ basis64
rel = ((recon - x64).norm() / x64.norm()).item()
print(f" rank-{ADALN_RANK} relative RMS residual: {rel:.3e} (reference impl: ~1.5e-5)")
adaln_basis = basis64.float().contiguous() # (8, 2688) fp32
adaln_mean = adaln_mean64.float().contiguous() # (2688,) fp32
adaln_t_table = c.float().contiguous() # (1025, 8) fp32
basisT = basis64.float().t().contiguous() # (2688, 8)
def fold_adaln(w, b, out_dtype):
"""Fold a released (out, 2688) AdaLN projection onto the 8-wide pruned coordinates."""
wf = w.to(torch.float32)
w_new = (wf @ basisT).to(out_dtype).contiguous() # (out, 8)
b_new = wf @ adaln_mean # (out,) -- the subspace offset
if b is not None:
b_new = b_new + b.to(torch.float32)
return w_new, b_new.contiguous()
# ---------------------------------------------------------------------------
# 2. Convert keys
# ---------------------------------------------------------------------------
print("\nConverting...")
dst = {
"time_embedder.table": adaln_t_table,
"adaln_basis": adaln_basis,
"adaln_mean": adaln_mean,
}
unmatched = []
def remap_attn(base_name):
"""Shared attention remap: fused qkv -> split, native names -> diffusers names."""
if ".attn.qkv_proj.weight" in base_name:
prefix = base_name.replace(".attn.qkv_proj.weight", ".attn.")
q, k, v = tensor.split(INNER_DIM, dim=0)
dst[f"{prefix}to_q.weight"] = q.contiguous()
dst[f"{prefix}to_k.weight"] = k.contiguous()
dst[f"{prefix}to_v.weight"] = v.contiguous()
return "handled"
if ".attn.qkv_proj.bias" in base_name:
prefix = base_name.replace(".attn.qkv_proj.bias", ".attn.")
q_b, k_b, v_b = tensor.split(INNER_DIM, dim=0)
dst[f"{prefix}to_q.bias"] = q_b.contiguous()
dst[f"{prefix}to_k.bias"] = k_b.contiguous()
dst[f"{prefix}to_v.bias"] = v_b.contiguous()
return "handled"
# Defensive: some H3 exports are unfused even in the refiner.
for native, target in (("q_proj", "to_q"), ("k_proj", "to_k"), ("v_proj", "to_v"), ("out_proj", "to_out.0")):
if f".attn.{native}.weight" in base_name:
return base_name.replace(f".attn.{native}.weight", f".attn.{target}.weight")
if f".attn.{native}.bias" in base_name:
return base_name.replace(f".attn.{native}.bias", f".attn.{target}.bias")
if ".attn.q_norm.weight" in base_name:
return base_name.replace(".attn.q_norm.weight", ".attn.norm_q.weight")
if ".attn.k_norm.weight" in base_name:
return base_name.replace(".attn.k_norm.weight", ".attn.norm_k.weight")
return None
for name, tensor in src.items():
new_name = None
if name in ("adaln_t_table", "adaln_basis", "adaln_mean"):
continue
if name.startswith("time_embedder."):
continue # replaced by the fitted table
if name == "video_patch_proj.weight":
new_name = "proj_in.weight"
elif name == "video_patch_proj.bias":
new_name = "proj_in.bias"
elif name == "audio_patch_proj.weight":
new_name = "audio_proj_in.weight"
elif name == "audio_patch_proj.bias":
new_name = "audio_proj_in.bias"
elif name == "condition_proj.weight":
new_name = "context_embedder.weight"
elif name == "condition_proj.bias":
new_name = "context_embedder.bias"
elif name == "rope.inv_freq":
print(" SKIP rope.inv_freq (rebuilt in __init__ from rope_freq_dim/rope_theta, non-persistent)")
continue
# --- final layer AdaLN: FOLD ---
elif name == "final_layer.adaln_proj.linear.weight":
w, b = fold_adaln(tensor, src.get("final_layer.adaln_proj.linear.bias"), tensor.dtype)
dst["norm_out.linear.weight"] = w
dst["norm_out.folded_bias"] = b
continue
elif name == "final_layer.adaln_proj.linear.bias":
continue
elif name == "final_layer.norm.weight":
new_name = "norm_out.norm.weight"
elif name == "final_layer.video_out.weight":
new_name = "proj_out.weight"
elif name == "final_layer.video_out.bias":
new_name = "proj_out.bias"
elif name == "final_layer.audio_out.weight":
new_name = "audio_proj_out.weight"
elif name == "final_layer.audio_out.bias":
new_name = "audio_proj_out.bias"
elif name == "token_refiner.final_norm.weight":
new_name = "token_refiner.final_norm.weight"
elif name.startswith("token_refiner.blocks."):
base_name = name.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
r = remap_attn(base_name)
if r == "handled":
continue
if r is not None:
new_name = r
elif ".mlp.fc1.weight" in base_name:
gate, value = tensor.chunk(2, dim=0) # swiglu: value | gate
new_name = base_name.replace(".mlp.fc1.weight", ".ff.net.0.proj.weight")
dst[new_name] = torch.cat([value, gate], dim=0).contiguous()
continue
elif ".mlp.fc1.bias" in base_name:
gate_b, value_b = tensor.chunk(2, dim=0)
new_name = base_name.replace(".mlp.fc1.bias", ".ff.net.0.proj.bias")
dst[new_name] = torch.cat([value_b, gate_b], dim=0).contiguous()
continue
elif ".mlp.fc2.weight" in base_name:
new_name = base_name.replace(".mlp.fc2.weight", ".ff.net.2.weight")
elif ".mlp.fc2.bias" in base_name:
new_name = base_name.replace(".mlp.fc2.bias", ".ff.net.2.bias")
else:
new_name = base_name
elif name.startswith("blocks."):
base_name = name.replace("blocks.", "transformer_blocks.", 1)
r = remap_attn(base_name)
if r == "handled":
continue
if r is not None:
new_name = r
elif ".mlp.fc1.weight" in base_name:
gate, value = tensor.chunk(2, dim=0)
new_name = base_name.replace(".mlp.fc1.weight", ".ff.net.0.proj.weight")
dst[new_name] = torch.cat([value, gate], dim=0).contiguous()
continue
elif ".mlp.fc1.bias" in base_name:
gate_b, value_b = tensor.chunk(2, dim=0)
new_name = base_name.replace(".mlp.fc1.bias", ".ff.net.0.proj.bias")
dst[new_name] = torch.cat([value_b, gate_b], dim=0).contiguous()
continue
elif ".mlp.fc2.weight" in base_name:
new_name = base_name.replace(".mlp.fc2.weight", ".ff.net.2.weight")
elif ".mlp.fc2.bias" in base_name:
new_name = base_name.replace(".mlp.fc2.bias", ".ff.net.2.bias")
# --- per-block AdaLN: FOLD ---
elif ".adaln_proj.linear.weight" in base_name:
w, b = fold_adaln(tensor, src.get(name.replace(".weight", ".bias")), tensor.dtype)
dst[base_name] = w
dst[base_name.replace(".adaln_proj.linear.weight", ".adaln_proj.folded_bias")] = b
continue
elif ".adaln_proj.linear.bias" in base_name:
continue
else:
new_name = base_name
unmatched.append(name)
else:
print(f" SKIP: {name}")
continue
if new_name is not None:
dst[new_name] = tensor.contiguous()
print(f"Converted keys: {len(dst)}")
# ---------------------------------------------------------------------------
# 3. Validation gate -- fail here, not after a 66 GB upload
# ---------------------------------------------------------------------------
print("\n--- validation ---")
assert tuple(dst["adaln_basis"].shape) == (ADALN_RANK, TIME_EMBED_DIM)
assert tuple(dst["adaln_mean"].shape) == (TIME_EMBED_DIM,)
assert tuple(dst["time_embedder.table"].shape) == (TIME_TABLE_SIZE, ADALN_RANK)
assert all(dst[k].dtype == torch.float32 for k in ("adaln_basis", "adaln_mean", "time_embedder.table"))
nw, nb = dst["norm_out.linear.weight"], dst["norm_out.folded_bias"]
assert tuple(nw.shape) == (NORM_OUT_ADALN_OUT, ADALN_RANK), f"norm_out not folded: {tuple(nw.shape)}"
assert tuple(nb.shape) == (NORM_OUT_ADALN_OUT,) and nb.dtype == torch.float32
layers = sorted({int(k.split(".")[1]) for k in dst if k.startswith("transformer_blocks.")})
assert len(layers) == NUM_LAYERS, f"expected {NUM_LAYERS} layers, got {len(layers)}"
for i in layers:
w = dst[f"transformer_blocks.{i}.adaln_proj.linear.weight"]
b = dst[f"transformer_blocks.{i}.adaln_proj.folded_bias"]
assert tuple(w.shape) == (BLOCK_ADALN_OUT, ADALN_RANK), f"block {i} not folded: {tuple(w.shape)}"
assert tuple(b.shape) == (BLOCK_ADALN_OUT,) and b.dtype == torch.float32, (i, b.shape, b.dtype)
for required in ("proj_in.weight", "audio_proj_in.weight", "context_embedder.weight",
"proj_out.weight", "audio_proj_out.weight", "norm_out.norm.weight"):
assert required in dst, f"missing {required}"
print(f"layers {len(layers)}")
print(f"norm_out {tuple(nw.shape)} w / {tuple(nb.shape)} folded_bias")
print(f"block 0 {tuple(dst['transformer_blocks.0.adaln_proj.linear.weight'].shape)} w / "
f"{tuple(dst['transformer_blocks.0.adaln_proj.folded_bias'].shape)} folded_bias")
print(f"AdaLN folds {sum(1 for k in dst if k.endswith('.adaln_proj.folded_bias'))} blocks + 1 norm_out")
if unmatched:
print(f"NOTE {len(unmatched)} keys hit the generic rename: {unmatched[:5]}")
# --- end-to-end: does the folded AdaLN reproduce the released AdaLN? ---
print("\n--- fold verification (released vs folded, float64) ---")
with torch.no_grad():
t_ck = torch.tensor([0.0, 0.137, 0.5, 0.873, 1.0], dtype=torch.float32)
x_ck = F.silu(time_embedder(time_proj(t_ck))).to(torch.float64) # (5, 2688)
c_ck = (x_ck - adaln_mean.double()) @ adaln_basis.double().t() # (5, 8)
def compare(tag, w_src_key, b_src_key, dst_w_key, dst_b_key):
Wr = src[w_src_key].double()
br = src[b_src_key].double()
ref = x_ck @ Wr.t() + br
got = c_ck @ dst[dst_w_key].double().t() + dst[dst_b_key].double()
err = ((got - ref).abs().max() / ref.abs().max()).item()
print(f" {tag:<22} max rel err = {err:.3e}")
assert err < 2e-2, f"{tag}: folded AdaLN does not reproduce the released one (err={err:.3e})"
compare("norm_out", "final_layer.adaln_proj.linear.weight", "final_layer.adaln_proj.linear.bias",
"norm_out.linear.weight", "norm_out.folded_bias")
compare("blocks.0.adaln_proj", "blocks.0.adaln_proj.linear.weight", "blocks.0.adaln_proj.linear.bias",
"transformer_blocks.0.adaln_proj.linear.weight", "transformer_blocks.0.adaln_proj.folded_bias")
print("validation OK")
# ---------------------------------------------------------------------------
# 4. Save + config
# ---------------------------------------------------------------------------
print(f"\nSaving to {LOCAL_DIR}...")
os.makedirs(LOCAL_DIR, exist_ok=True)
save_file(dst, os.path.join(LOCAL_DIR, "diffusion_pytorch_model.safetensors"))
print("Fetching config...")
base_repo_path = snapshot_download(
"multimodalart/MiniMax-H3-Pruned",
allow_patterns=["transformer/config.json", "transformer/modeling_minimax_h3_pruned.py"],
local_dir="./_temp_base_repo",
)
with open(os.path.join(base_repo_path, "transformer", "config.json")) as f:
config = json.load(f)
config["_class_name"] = "MiniMaxH3PrunedTransformer3DModel"
with open(os.path.join(LOCAL_DIR, "config.json"), "w") as f:
json.dump(config, f, indent=2)
src_modeling = os.path.join(base_repo_path, "transformer", "modeling_minimax_h3_pruned.py")
if os.path.exists(src_modeling):
shutil.copy2(src_modeling, os.path.join(LOCAL_DIR, "modeling_minimax_h3_pruned.py"))
shutil.rmtree("./_temp_base_repo", ignore_errors=True)
# ---------------------------------------------------------------------------
# 5. Load smoke test BEFORE uploading
# ---------------------------------------------------------------------------
print("\nSmoke-testing the local model...")
# `AutoModel.from_pretrained` cannot take `output_loading_info=True` here -- it treats the returned
# (model, info) tuple as the model and crashes on `model._diffusers_load_id`. Load plainly (this already
# validates every tensor shape and raises on any mismatch), then diff the two key sets ourselves.
from safetensors import safe_open
WEIGHTS = os.path.join(LOCAL_DIR, "diffusion_pytorch_model.safetensors")
model = AutoModel.from_pretrained(LOCAL_DIR, trust_remote_code=True)
with safe_open(WEIGHTS, framework="pt") as f:
file_keys = set(f.keys())
model_keys = set(model.state_dict().keys()) # persistent buffers only -> rope.inv_freq is absent from both
missing = sorted(model_keys - file_keys) # the model expects these; the file does not carry them
unexpected = sorted(file_keys - model_keys) # in the file but ignored by the model
print(f"missing : {missing[:10]}{' ...' if len(missing) > 10 else ''}")
print(f"unexpected : {unexpected[:10]}{' ...' if len(unexpected) > 10 else ''}")
if missing or unexpected:
raise SystemExit("key sets do not match -- fix before uploading.")
# ---------------------------------------------------------------------------
# 6. Upload
# ---------------------------------------------------------------------------
print(f"\nUploading to {REPO_ID}...")
api = HfApi(token=HF_TOKEN)
create_repo(REPO_ID, exist_ok=True, private=True)
api.upload_folder(folder_path=LOCAL_DIR, repo_id=REPO_ID, repo_type="model")
print(f"\nDone! Uploaded to {REPO_ID}")