ECAPA-TDNN speaker embedding (3D-Speaker), in ONNX
A speaker-embedding model in ONNX: it turns 3.6 seconds of speech into a vector of 192 numbers that is close to the vector of the same
speaker and far from that of another. It is a conversion, made by MPAI - Moving Picture, Audio and Data Coding by Artificial
Intelligence - of the ECAPA-TDNN of the 3D-Speaker project, for Thalia, the avatar of
MPAI as a Service, whose Apps MAC and ACR (access control and registration) recognise a person by voice with it. The software is at
https://github.com/mpai-community/mpai-sw, in AIMs/MMC/V2.5/SIR. It is not an official release of 3D-Speaker.
Where it comes from - and what is not certain
The model is the ECAPA-TDNN of 3D-Speaker (192-dimensional embeddings from 80-dimensional filterbank features, 16 kHz), most probably the
checkpoint iic/speech_ecapa-tdnn_sv_zh-cn_3dspeaker_16k
(ecapa-tdnn.ckpt), whose training data and results are described on its page.
The exact checkpoint and the conversion command of this file were not recorded. Converting that published checkpoint again with
3D-Speaker's own recipe (docs/conversion/export_ecapa_tdnn.py in the repository above) gives embeddings that are nearly, but not exactly, the
same as the ones of this file (cosine similarity 0.998 to 0.9999 on test inputs). If you need a file you can reproduce, make it with that script.
The file
| File | Size (bytes) | SHA-256 |
|---|---|---|
ecapa-tdnn.onnx | 83,303,532 | 38FDFC7D2BC9E2925349BAAB9639FCAF4B4C755BE83EE7D616B6E3FBC9D5EAB3 |
Input feature, float32, [1, 360, 80]: 360 frames (3.6 s) of 80-dimensional log-Mel filterbank features. The length is fixed.
Output embedding, float32, [1, 192]. ONNX opset 11.
The features it expects
Kaldi-style filterbank, as in the 3D-Speaker recipes: 16 kHz mono; 25 ms Povey window, 10 ms hop; DC offset removed, pre-emphasis 0.97;
80 mel bins from 20 Hz to 7600 Hz; natural-log energies; mean normalisation over the utterance. Speech shorter than 360 frames is padded and
longer speech truncated. AIMs/MMC/V2.5/SIR/MelSpectrogram.cs in the repository above is an implementation in C#. Two embeddings are compared
by their cosine similarity; the threshold above which two voices are taken to be the same person depends on your use and has to be measured with your own speakers.
Limitations and responsible use
- A voice is biometric personal data. Under the GDPR and similar laws, its use for identifying a person needs a legal basis, a notice, and care in how embeddings are stored.
- It is not a security mechanism: a recording, or a voice made by a program, can be taken for a person. Do not use it alone to give access to anything important.
- Its accuracy depends on the language, the microphone, the noise and how long the person speaks, and on who was in the training data; measure it on your own speakers.
- Fixed input length: 3.6 s.
Licence and credit
Apache License 2.0, as the 3D-Speaker code and model: the licence text is in LICENSE. This is a modified version of the original
(converted to ONNX with a fixed input length); the changes are stated here, as the licence requires. Credit for the model: the 3D-Speaker team,
Speech Lab, Alibaba Group. Cite:
@article{chen20243d,
title = {3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization},
author = {Yafeng Chen and Siqi Zheng and Hui Wang and Luyao Cheng and Tinglong Zhu and Rongjie Huang and Chong Deng and Qian Chen and Shiliang Zhang and Wen Wang and Xihao Li},
journal = {arXiv preprint arXiv:2403.19971},
year = {2024}
}
The conversion, and this file, are by MPAI (https://mpai.community).