Acoustic Token Admixture: model weights and speaker pool
Pretrained checkpoints and the pseudospeaker pool for the Interspeech 2026 paper "Acoustic token admixture for joint speaker and content anonymization" (Ali Golmakani, Seyed Ahmad Hosseini, Omar Manil Bendali, Emmanuel Vincent, Brij Mohan Lal Srivastava).
- Code: https://github.com/Nijta/acoustic-token-admixture
- Audio samples: https://nijta.github.io/acoustic-token-admixture/
The system anonymizes both the speaker's voice and named entities in the content. It mixes encoder-derived and phoneme-conditioned RVQ tokens frame by frame, then resynthesizes the result with a BigVGAN vocoder conditioned on a pseudospeaker. On VoicePrivacy 2024 it reaches EER 42.54% with WER 3.73% (β = 0.7).
Quick start
git clone https://github.com/Nijta/acoustic-token-admixture.git
cd acoustic-token-admixture
pip install -r requirements.txt
hf download nijta/acoustic-token-admixture --local-dir models
(cd models && sha256sum -c SHA256SUMS)
export MODELS_DIR=$PWD/models
export PYTHONPATH=".:./audiolm:./bigvgan/src"
python static_infer.py --inputs input.wav --output-dir outputs --seeds 52 \
--gender any --admixture-ratio 0.7 --pitch-f 0.75 --db 2
The folder layout matches what the code expects in $MODELS_DIR.
Files
| Path | Component | Paper |
|---|---|---|
RVQWhisper/rvq_model_en.pth, config.yaml | 8 x 1024 residual vector quantizer on frozen Whisper large-v2 encoder features (Stream A) | Sec. 2.1.1 |
Aligner/aligner_english.pth | CTC phoneme head (111 classes) for forced alignment | Sec. 2.1.2 |
Aligner/dur_predictor.pt, f0_predictor.pth | Duration and F0 predictors used for content replacement | Sec. 2.3 |
AudioLM/audiolm_english.pt, config.yaml | Autoregressive decoder from articulatory features to RVQ indices (Stream B) | Sec. 2.1.2 |
BigVGAN/generator_english, config.json | BigVGAN generator conditioned on RVQ reconstruction, F0 and x-vector (1473 dims, 16 kHz) | Sec. 2.4 |
POOL/english/ | Pseudospeaker pool: ECAPA-TDNN x-vectors, gender map, Affinity Propagation cluster indices, pitch statistics | Sec. 2.4 |
*_french*, rvq_model_fr.pth, POOL/french/ | French counterparts of the models and pool above | Not evaluated in the paper |
SHA256SUMS | Checksums for every file |
The French checkpoints are provided as is. The paper evaluates English only, and the command-line tools in the code repository load the English models by default.
Training data
- RVQ-Whisper: Common Voice 17, LibriTTS, TED-LIUM and VoxPopuli (English), filtered with wv-mos > 3.3 and < 10 min per speaker.
- Token decoder and BigVGAN: Common Voice 17, LibriTTS train-clean-100, VoxPopuli, VCTK and LibriSpeech train-clean-360 for English. The French models use Common Voice 17, Multilingual LibriSpeech, VoxPopuli and VoxForge.
- Speaker pools: ECAPA-TDNN x-vectors of Common Voice speakers held out from all evaluation sets (English: 1,465 speakers; French: 251 speakers). Speaker IDs are Common Voice's hashed client IDs. No audio or transcripts are included.
License
The weights and the speaker pool are released under CC BY-NC 4.0: you may use them for research and other non-commercial purposes, with attribution. Some of the training corpora carry non-commercial terms. The code is MIT-licensed; see the GitHub repository. For commercial use, contact Nijta.
Intended use and limitations
These models are for protecting speakers when recordings are shared or reused, and for research on speech privacy.
- Do not use them to impersonate people, or to make synthetic speech look like a real person's recording.
- Do not try to re-identify the speakers behind the pool x-vectors.
- The content protection only covers what the NER tagger finds. Review anonymized data before release.
- Emotion preservation is limited (UAR 40.11% on IEMOCAP).
Citation
@inproceedings{golmakani2026acoustic,
title = {Acoustic token admixture for joint speaker and content anonymization},
author = {Golmakani, Ali and Hosseini, Seyed Ahmad and Bendali, Omar Manil and Vincent, Emmanuel and Srivastava, Brij Mohan Lal},
booktitle = {Proc. Interspeech 2026},
year = {2026}
}