AudioSeg
AudioSeg predicts topic boundaries directly from audio, on a 6-second grid, and was introduced in our paper Beyond Transcripts: A Renewed Perspective on Audio Chaptering (acl | arXiv). Given an audio file (e.g. a podcast, lecture, or YouTube video), the model outputs a boundary probability for every 6-second segment, which can be thresholded into chapter/topic boundaries.
Architecture
- Encoder (frozen):
openai/whisper-large-v3encoder, applied to consecutive 30-second chunks of the audio. - Head (trained, ~32M params):
- a local segment transformer that pools encoder frames into one embedding per 6-second segment via a learned
[SEG]token - a document encoder over the segment sequence producing per-segment boundary probabilities
- a local segment transformer that pools encoder frames into one embedding per 6-second segment via a learned
Only the head weights are stored in this repository; the Whisper encoder is downloaded from openai/whisper-large-v3 on first use.
Usage
pip install transformers torch torchaudio
from transformers import AutoModel
model = AutoModel.from_pretrained("retkowski/audioseg", trust_remote_code=True)
model = model.to("cuda")
result = model.segment("episode.mp3")
result["ts_boundaries"] # boundary timestamps in seconds, e.g. [315.0, 747.0, ...]
result["probs"] # boundary probability per 6s segment
result["segment_size_sec"] # 6.0
segment() also accepts a waveform tensor (model.segment(waveform, sample_rate=sr)), a custom decision threshold (default 0.5), and batch_chunks to control how many 30-second chunks are encoded per Whisper batch.
A reported timestamp t is the midpoint of the 6-second segment in which a topic change was detected, i.e. the change lies within [t - 3, t + 3) seconds. The start of the audio is not considered a boundary, so the first segment is never reported.
Training
The model was trained on YTSeg, YouTube videos with human-created chapter markers, using the chapter start times as boundary supervision (binary cross-entropy on the 6-second grid). The Whisper encoder was kept frozen.
Citing
We kindly request you to cite our corresponding paper if you use our model:
@inproceedings{retkowski-etal-2026-beyond,
title = "Beyond Transcripts: A Renewed Perspective on Audio Chaptering",
author = {Retkowski, Fabian and
Z{\"u}fle, Maike and
Nguyen, Thai Binh and
Niehues, Jan and
Waibel, Alexander},
editor = "Liakata, Maria and
Moreira, Viviane P. and
Zhang, Jiajun and
Jurgens, David",
booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
month = jul,
year = "2026",
address = "San Diego, California, United States",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.acl-long.396/",
doi = "10.18653/v1/2026.acl-long.396",
pages = "8765--8787",
ISBN = "979-8-89176-390-6",
abstract = "Audio chaptering, the task of automatically segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and videos. Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation. We address these gaps through three contributions: (1) a systematic comparison between text-based models with acoustic features, a novel audio-only architecture (AudioSeg) operating on learned audio representations, and multimodal LLMs; (2) empirical analysis of factors affecting performance, including transcript quality, acoustic features, duration, and speaker composition; and (3) formalized evaluation protocols contrasting transcript-dependent text-space protocols with transcript-invariant time-space protocols. Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and current MLLMs struggle due to context limitations and weak instruction following."
}
License
CC-BY-NC-4.0 (non-commercial use).