MSP-AVSR
MSP-AVSR is the multimodal model in Multimodal Speech Perception (MSP). It combines Wav2Vec2 audio representations and AV-HuBERT visual representations through bidirectional cross-attention, then predicts English text with CTC.
| Property | Value |
|---|---|
| Inputs | Synchronized mono 16 kHz audio and visible-speaker video |
| Audio / visual encoders | MSP-ASR / MSP-VSR |
| Fusion | Mask-aware, 12-head bidirectional cross-attention |
| Fusion / feed-forward width | 768 / 3,072 |
| Demo | MSP-AVSR |
Architecture
- MSP-ASR and MSP-VSR encode audio and mouth-region video independently.
- Separate projections and layer normalization map both sequences to a shared 768-dimensional space.
- Two cross-attention paths model both directions: audio queries attend to visual keys/values, while visual queries attend to audio keys/values. Each path uses padding masks, a residual connection, and layer normalization.
- The visual-query output is upsampled to the audio frame rate, concatenated with the audio-query output, projected, and processed by a 3,072-dimensional GELU feed-forward block.
- A shared CTC head predicts the transcript. Training adds auxiliary audio and visual CTC losses to the fused loss with weights
0.5 / 0.5 / 1.0.
In evaluation mode, the checkpoint can also process audio-only or video-only inputs through the corresponding pretrained branch.
Usage
import torch
from transformers import AutoModelForCTC, AutoProcessor
model_id = "MahmoodAnaam/MSP-AVSR"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCTC.from_pretrained(model_id, trust_remote_code=True).eval()
inputs = processor(
audio="sample.mp4",
videos="sample.mp4",
return_tensors="pt",
)
with torch.inference_mode():
logits = model(**inputs).logits
text = processor.tokenizer.batch_decode(logits.argmax(dim=-1))[0]
print(text)
Audio and video must represent the same time interval. For 3-gram beam search, install pyctcdecode and kenlm, then decode with MahmoodAnaam/MSP-Processor-With-LM. Pin Hub revisions when loading custom code in controlled environments.
Training
| Setting | Value |
|---|---|
| Maximum steps | 30,000 |
| Learning rate | 1e-4 |
| Train / eval batch size per device | 32 / 32 |
| Scheduler | Cosine; 1,000 warmup steps |
| Precision | bfloat16 |
| Audio / visual / fused CTC weights | 0.5 / 0.5 / 1.0 |
The selected validation checkpoint recorded loss 1.1193 and WER 17.36%. The recorded stack was Transformers 5.10.2, PyTorch 2.10.0 with ROCm 7.2.4, Datasets 4.0.0, and Tokenizers 0.22.2.
Evaluation
The following tables report MSP-AVSR only. Results use MSP-Processor-With-LM; WER is a percentage and lower is better.
LRS2
| Evaluation | WER |
|---|---|
| Clean test | 3.93 |
| Clean + eight noisy conditions, macro average | 22.80 |
| Interferer | -5 dB | 0 dB | 5 dB | 10 dB |
|---|---|---|---|---|
| 1 | 48.85 | 20.90 | 22.41 | 12.33 |
| 2 | 40.92 | 19.91 | 22.86 | 13.08 |
AVCocktail
| Segmentation | WER |
|---|---|
| Active-speaker-detection chunks | 67.73 |
| Fixed 10-second chunks | 107.46 |
| Gold chunks | 63.64 |
AVCocktail values are word-weighted across 51 videos. WER can exceed 100% when insertions are numerous. Scores depend on preprocessing, synchronization, face tracking, dataset revision, checkpoint, segmentation, and decoder settings.