Surround Sound: speech event detector

Demo: Surround Sound Space: a 59-minute documentary with the detected sounds, per-tag probabilities and an aligned transcript, synced to the video.

A small detector (5.8M parameters) that finds non-verbal vocal sounds (breath, cough, laugh, sigh, sniff, throat clearing, …) and background sounds (music, horn, vehicle, dog bark, siren, rain, applause, …) in recordings, with start and end times. It covers 54 classes. The included code also merges detected events into a Whisper transcript. An illustrative example:

so I was saying [cough] that the <horn> road was blocked </horn> [breath] anyway

[tag] marks a sound made by the speaker, placed where it starts. <tag> … </tag> wraps the words a background sound overlaps.

• Input: audio of any length (any format ffmpeg reads), resampled to 16 kHz mono.
• Output: a list of {label, start, end, prob} events, one probability per class every 40 ms, or a transcript with tags.
• Speed: about 39× real time on 8 CPU threads and 380× on an NVIDIA T4 (3 minutes of audio).

Quick start

pip install torch torchaudio numpy scipy safetensors huggingface_hub    # plus the ffmpeg binary
pip install openai-whisper                                               # optional, for transcripts
hf download theboringai-work/surround-sound --local-dir surround-sound
cd surround-sound
python inference.py my_audio.mp3                    # transcript with tags + event list
python inference.py my_audio.mp3 --asr none         # events only
python inference.py recordings/ --asr none --json events.jsonl

From Python, run from the downloaded folder:

from sed import Detector, rich_transcript, clean

det = Detector.from_pretrained("theboringai-work/surround-sound", profile="sensitive")   # or a local folder path
events = det.detect("my_audio.mp3")
# [{'label': 'cough', 'start': 0.68, 'end': 2.0, 'prob': 0.92}, {'label': 'breath', 'start': 7.72, 'end': 8.72, 'prob': 0.94}, ...]

times, probs = det.frame_probs(audio_16k)          # raw per-class probabilities on the 40 ms grid

# with any ASR that gives word timestamps: words = [(word, start_s, end_s), ...]
print(rich_transcript(words, clean(words, events)))

Useful options of inference.py:

Files

Model

audio 16 kHz ─► log-mel (64 bands, 25 ms window, 10 ms hop)
            ─► 5 conv blocks (3×3, BatchNorm, ReLU; frequency /16, time /4)
            ─► linear projection + sinusoidal positions
            ─► Transformer encoder (6 layers, d=256, 8 heads, pre-norm, GELU)
            ─► linear + sigmoid: 54 probabilities every 40 ms

The model sees 8 s windows. Long audio is processed with a 2 s hop, and overlapping predictions are averaged. Each class's probability track is then median-filtered over 200 ms and thresholded with hysteresis (an event starts above the class threshold and continues while above 0.6 × threshold). Events closer than 0.3 s (speaker sounds) or 1.0 s (background) are joined, and events shorter than 0.15 s are dropped.

Threshold profiles

Every class has its own threshold, tuned on real held-out recordings by 2-fold cross-validation.

Classes and accuracy

frame AP: frame-level average precision on real held-out windows. CV F1: cross-validated frame F1 at the balanced threshold. found: share of held-out real events detected (overlapping prediction of the same class) with the sensitive profile. weak: too little held-out data (< 1 s of positive frames) or CV F1 < 0.1; treat these tags as hints.

Speaker sounds, [tag]

Background sounds, <tag> … </tag>

Evaluation

All evaluation uses real recordings held out from training:

• NonVerbalSpeech-38K: a 10% hash split.
• Vaani Noise Event Timestamps: verified clips from 14 held-out districts.
• NonverbalTTS: dev + test.
• AudioSet-Strong: the evaluation split.

Each class is scored only on the datasets that label it. Whole clips are run through the full detector; a true event is found if a same-class prediction overlaps it, and a prediction is correct if it overlaps a same-class true event. Macro averages over classes with ≥ 10 held-out events:

Thresholds were tuned on held-out data that overlaps the evaluation clips, so absolute numbers are somewhat optimistic. No standard benchmark (e.g. DCASE) has been run.

Training data

Real annotated recordings, event crops pasted into speech at known times (synthetic mixtures), and augmentation with real background noise.

Training examples (171,000 windows of 8 s, including augmented copies) were drawn from about 168 h of real annotated training recordings, plus event crops, speech beds and background noise; held-out recordings (about 17 h) were used only for model selection, thresholds and evaluation. Training ran for 10 epochs on one NVIDIA T4 (about 18 minutes per epoch), warm-started from a preliminary 31-class checkpoint of the same architecture. The checkpoint with the best mean macro average precision on the real held-out sets is kept (epoch 6).

Limitations and intended use

• Intended for research and non-commercial use: rich transcription, corpus analysis, data filtering, and annotation assistance with human review.
• Domains: training audio is mostly Mandarin drama, Indic phone speech, English interviews and acted speech, and YouTube. Accuracy on other domains (e.g. call centres, meetings, far-field microphones) is unmeasured. Check it on your own labelled sample.
• Weak classes (tv_speaker, fan, footsteps, cry, groan, chew, door, yawn, hiccup, nose_blow, knock) fire rarely and unreliably.
• Steady hum from some microphones can trigger machine or fan. Use --hide machine,fan if needed.
• Timing: tag positions in transcripts depend on the ASR's word timestamps.
• Not suitable for health or diagnostic use (e.g. inferring illness from coughs), surveillance, or any decision about a person.

License

The weights are released under CC BY-NC-SA 4.0 to respect the non-commercial and share-alike terms of the training data above. One source (OpenSLR 99) is CC BY-NC-ND 4.0. Users are responsible for complying with the terms of the training data in their jurisdiction. Commercial use is not permitted.

Citation

@misc{surround_sound_2026,
  title  = {Surround Sound: speech event detector},
  author = {Bar, Ayush Kumar},
  year   = {2026},
  note   = {Model on the Hugging Face Hub},
  url    = {https://huggingface.co/theboringai-work/surround-sound}
}

Please also cite the training datasets (see their pages).