AV-STE: Audio-Visual Speech Token Enhancement
EMNLP 2026 — Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
Code, demo, and full documentation: github.com/bellagodiva/av-ste
AV-STE recovers clean Mimi semantic tokens from noisy speech by fusing audio features with lip-ROI video through an entropy-gated cross-attention module built on top of AV-HuBERT. Enhanced tokens can be fed directly into any Mimi-based TTS or codec LM (e.g. Moshi) in place of the noisy ones.
Files
| Checkpoint | Use for |
|---|---|
avste.pt | Clean speech, non-speech background noise, cross-dataset speaker interference |
avste_lrs3_interference.pt | Same-corpus-style competing speakers, out-of-domain video |
Together they reproduce every semantic-token-accuracy row in the paper's main table. large_vox_iter5.pt (the public AV-HuBERT-Large VoxCeleb2 backbone both were fine-tuned from) is not hosted here — the GitHub repo's download script fetches it directly from its original source.
Full checkpoint-selection guidance: checkpoints/README.md in the code repo.
Usage
from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="bgdv99/av-ste", filename="avste.pt")
# or filename="avste_lrs3_interference.pt"
Both are fairseq model ensemble files:
from fairseq import checkpoint_utils
models, cfg, task = checkpoint_utils.load_model_ensemble_and_task(
["avste.pt"],
arg_overrides={"w2v_path": "large_vox_iter5.pt"},
)
See the GitHub repo for the full inference pipeline (scripts/infer_avste.py), a runnable demo, and training instructions.
Data
These weights were fine-tuned on LRS3 (license-gated) with AudioSet noise/interference augmentation. No training or evaluation data is redistributed here — only the resulting model parameters.
Citation
@inproceedings{avste2026,
title = {Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models},
author = {TODO},
booktitle = {Proceedings of EMNLP},
year = {2026},
}