QaDiT — text-to-audio latent Diffusion Transformer

A ~159M-parameter latent Diffusion Transformer that turns a text caption into 10.24 s of 16 kHz mono audio: FLAN-T5 conditioning → DiT denoising of AudioLDM KL-VAE latents → VAE decode → HiFi-GAN vocoder.

Example: Prompt: A small waterfall flows through a forest while insects buzz and birds sing.

Output:

1. Big picture — three pipelines

2. Offline pre-compute (frozen models, run once)

Heavy frozen models run once; the training loop never loads T5, the VAE, or the REPA encoder.

3. Training step (what is actually optimized)

Only the DiT and its small glue layers receive gradients.

4. Inside one DiT block

5. Inference / sampling (caption → waveform)

6. Component ownership

Training objective (DDIM + v-prediction)

Forward process

$$ z_t = \sqrt{\bar{\alpha}_t},z_0

• \sqrt{1-\bar{\alpha}_t},\varepsilon $$

Network target

$$ v = \sqrt{\bar{\alpha}_t},\varepsilon

• \sqrt{1-\bar{\alpha}_t},z_0 $$

At sample time the DiT predicts v; we recover \hat{z}_0 and \hat{\varepsilon}, then step with DDIM (\eta = 0). CFG is applied in v-space with default scale (s = 4.0). After DDIM, latents are divided by latent_scale ≈ 0.95035 before VAE decode — that whole chain is what model.generate() runs.

Dataset

AudioCaps is captioned environmental / everyday sound — not speech or music. Those domains are out of distribution for this checkpoint.

Training run (this checkpoint)

Training curves

Put W&B / TensorBoard screenshots (or exports) under assets/ using the filenames below. Until then the images show as broken links on the Hub — that is intentional so the slots are obvious.

Diffusion / total loss

REPA loss

Usage

pip install transformers diffusers soundfile sentencepiece

import soundfile as sf
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("QuarkML/QaDiT", trust_remote_code=True)
model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval()

out = model.generate(
    "A small waterfall flows through a forest while insects buzz and birds sing.",
    num_inference_steps=200,
    guidance_scale=16.0,
    seed=0,
)
sf.write("sample.wav", out.audios[0], out.sampling_rate)

First generate downloads the frozen helpers this run was trained with: google/flan-t5-large and the VAE + vocoder from cvssp/audioldm-s-full-v2.

Output types

Precomputed T5 states

out = model.generate(
    encoder_hidden_states=text_emb,      # [B, 64, 1024]
    encoder_attention_mask=text_mask,    # [B, 64]
)

Single denoising step

v = model(latents, timesteps, encoder_hidden_states, encoder_attention_mask).sample

Precision and devices

Runs on CPU and CUDA. With dtype=torch.float16 or torch.bfloat16 the DiT backbone runs in half precision; DDIM schedule math stays in float32. Keep T5 / VAE / vocoder in float32 (FLAN-T5 overflows easily in fp16).

Important details

• config.latent_scale (0.9503493) must match training precompute. generate divides by it before VAE decode.
• Every sample is fixed length: 10.24 s @ 16 kHz.
• repa_layer exists for REPA fine-tuning; inference ignores it.
• Sampling always uses the EMA weights packaged here.

Limitations and intended use

Intended use: education, reproduction of a small latent DiT audio stack, ablations, and a starting checkpoint for longer / wider training.

Not intended for: production SFX libraries, speech synthesis, music generation, or safety-critical audio.

Known limits of this checkpoint

• ~24k steps on ~45k AudioCaps clips — undertrained vs public SOTA systems
• Weak on speech, music, and densely described scenes
• Inherits caption biases and coverage holes of AudioCaps
• Prefer the default 50 DDIM steps for demos; low step counts sound coarse

Research artifact — how to improve this

This release is a research artifact, not a production host model. The architecture and sampling path are solid enough to build on; the ceiling is mostly data and compute:

1. Train longer — continue past 24k steps with the same recipe (or lower LR).
2. Scale the dataset — mix in larger captioned audio corpora beyond AudioCaps.
3. Retune sampling — CFG scale, DDIM step count, and prompt formatting.
4. Keep measuring — log diffusion loss, REPA loss, and listening tests.

Those levers will move quality more than inventing a new backbone for this size of model. Contributions and longer runs are welcome; treat this Hub page as a reproducible baseline, not a finished product.

Citation

@misc{qadit2026,
  title        = {QaDiT: A Text-to-Audio Latent Diffusion Transformer},
  author       = {Sidharth GN},
  year         = {2026},
  note         = {Research artifact. Weights and transformers remote-code loading.}
}