DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

Paper (arXiv:2608.09288) · Code (GitHub)

DAVE performs audio-visual speech enhancement and target speaker extraction on real-world meeting recordings. An audio-only separation backbone (TIGER-M, 2.56M parameters) reconstructs two anonymous speech streams; a four-cue weighted-fusion module assigns speaker identities from face video; a selective enhancement chain finalizes output. Because the backbone never consumes video, degraded visual input cannot corrupt the reconstructed audio.

🏆 Challenge Results (official leaderboard, seven-column rank-mean)

Track 1 — 4th of 17 teams (top 5 shown):

Track 2 — 3rd of 13 teams (top 5 shown):

Model Architecture

Training

A six-term loss, introduced as a curriculum (one term at a time, verified for no-regression before its weight is raised):

L = L_SI-SDR + λ_cer·L_CER + λ_spk·L_spk + λ_utmos·L_UTMOS + λ_stoi·L_STOI + λ_pesq·L_PESQ

Training data: the open 12,000-pair subset is released as DAVE-Corpus (sources: AliMeeting / AISHELL-4 / MUSAN, CC BY-SA 4.0). The full training pool additionally used subsets built from MISP-Meeting and official challenge dev data, which cannot be redistributed — see DATA_LICENSE.

Training code and the data-synthesis pipeline: GitHub — TaurenMountain/DAVE.

Quick Start

Separation is self-contained — the TIGER-M model code is bundled under src/models/, so only torch, numpy, soundfile are needed:

pip install torch numpy soundfile

python inference.py --mixture samples/echoset_01/mix.wav --out output
python inference.py --mixture path/to/dir --out output --device cuda

Output: output/<name>/s1.wav, s2.wav (16 kHz mono). Input must be 16 kHz. samples/echoset_01/ contains a playable example pair (mix / s1 / s2, from the Apache-2.0 EchoSet subset).

Python API

import sys, torch, soundfile as sf
sys.path.insert(0, "src")
from models import TIGER

model = TIGER(sample_rate=16000, in_channels=512, out_channels=256,
              num_blocks=12, upsampling_depth=5, win=640, stride=160, num_sources=2)
state = torch.load("model/real_avse_tiger_m.ckpt", map_location="cpu")["state_dict"]
model.load_state_dict(state, strict=False)
model.eval()

mix, sr = sf.read("samples/echoset_01/mix.wav", dtype="float32")  # 16 kHz mono
with torch.no_grad():
    est = model(torch.from_numpy(mix[None]))
sf.write("s1.wav", est[0, 0].numpy(), 16000)
sf.write("s2.wav", est[0, 1].numpy(), 16000)

Files

• model/real_avse_tiger_m.ckpt — TIGER-M weights (2.56M parameters), the challenge submission checkpoint;
• inference.py + src/models/ — self-contained separation CLI;
• samples/ — one example pair for format inspection.

Training code, the data-synthesis + transcription pipeline, and reference implementations of speaker attribution / enhancement / evaluation live in the GitHub repository.

Citation

@misc{zhou2026dave,
  title         = {{DAVE}: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation},
  author        = {Zhou, Wei and Ning, Wanyi and Guo, Yinshang and Fang, Qianxiao and Qian, Haitao and Li, Yingpeng},
  year          = {2026},
  eprint        = {2608.09288},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2608.09288}
}