🎬 Demo Video
Please visit our ✨demopage✨ .
🏆 SoulX-Transcriber Overview
📖 Introduction
SoulX-Transcriber is a unified end-to-end large audio language model for multi-speaker diarization and recognition in multi-speaker dialogue scenarios. Rather than relying on a cascaded pipeline, the model directly learns speaker attribution, timestamped segmentation, and transcription in a single framework, producing coherent speaker-consistent transcripts for overlapping and fast-turn conversations.
🌟 Highlights
- State of the art performance. SoulX-Transcriber achieves superior performance on the AISHELL-4 and AliMeeting benchmarks via a unified diarization and recognition framework, which directly produces structured outputs consisting of timestamps, speaker labels, and transcripts.
- Speaker-aware multi-stage training. Speaker-aware multi-task Continues Pre-Training plus Supervised Fine-tuned strengthens speaker representation and robustness to conversations, mitigating same-gender confusion, overlap, and boundary errors.
- A more natural and authentic approach to dialogue generation. We propose a speaker characteristics-driven audio matching pipeline that automatically selects the most suitable reference audio for each utterance, producing more natural, context-aligned simulated dialogues.
📊 Results
Utterance-level Evaluation on open-source datasets
Segmented Evaluation (5 minutes segments)
† Closed-source model. DER unavailable for SpeakerLM and SOT-based models due to incompatible output formats.
Internal Multi-domain Evaluation
🧪 Multi-speaker Dialogue Simulation Pipeline
To improve out-of-domain generalization, we build an agent-based multi-speaker dialogue simulation pipeline with a speaker-aware prompt audio matching mechanism. Given a target dialogue text, the system analyzes speaker tags, selects the most suitable reference audio for each speaker using multi-dimensional speaker representations, and synthesizes context-consistent multi-turn dialogue audio.
Workflow: building dialogue text database → building reference audio database → target text analysis → reference audio matching → dialogue audio generation. Detailed information is shown on the figure below.
-
Dialogue text database. We collect multi-speaker dialogue texts from Chinese/English podcasts and novels. An LLM annotates speaker tags and controls the number of speakers; we keep segments with 3–8 speakers to ensure natural, coherent dialogue context.
-
Dialogue context analysis: We use Qwen3-8B as the LLM brain for speaker-tag and context analysis, and MOSS-TTSD for long-form, multi-speaker multi-turn TTS synthesis.
-
Reference audio database. We run VAD on long-form drama audio, cut it into 3–10s clips, and filter by UTMOS and SNR to ensure quality. Each clip is annotated by Gemini-3.1-pro-preview with multi-dimensional speaker attributes (e.g., gender/age/emotion/speech rate/pitch/timbre/style/tone/role state). We embed each attribute using bge-m3 and stack them into a per-clip feature matrix, forming an embedding index for retrieval.
-
Best reference–audio matching. Given a target dialogue with speaker tags, an LLM analyzes each speaker’s attributes and builds the same multi-dimensional embedding representation. We compute similarity against all reference clips, apply a weighted score across attribute dimensions, and retrieve top-k (k=3) candidates per speaker. A final selection enforces diversity (different source speakers) and UTMOS consistency (|Δ| ≤ 0.5) to produce the best reference audio set for synthesis.
Installation
Environment Setup
git clone https://github.com/Soul-AILab/SoulX-Transcriber.git
cd SoulX-Transcriber
conda create -n soulx_transcriber python=3.12 -y
conda activate soulx_transcriber
Install MS-Swift and dependencies:
pip install ms-swift
Training & Fine-tuning
SoulX-Transcriber shares the same architecture with Qwen3-Omni-30BA3B-Instruct. We recommend users conduct continued pre-training and fine-tuning for this model via the ms-swift toolkit.
Inference
vLLM-omni
SoulX-Transcriber is built on top of Qwen3-Omni-30B-A3B-Instruct. We recommend using vllm-omni for inference..
cd your_env_path/
# install uv：
curl -LsSf https://astral.sh/uv/install.sh | sh
# create new uv environment（using aliyun mirror）
uv venv vllm_omni --python 3.12 --seed --index-url https://mirrors.aliyun.com/pypi/simple/
# activate uv environment
source vllm_omni/bin/activate
# install vllm：
uv pip install vllm --torch-backend=auto --index-url https://mirrors.aliyun.com/pypi/simple/
# install vllm-omni:
uv pip install vllm-omni --index-url https://mirrors.aliyun.com/pypi/simple/
# install gradio (Optional)：
uv pip install 'vllm-omni[demo]' --index-url https://mirrors.aliyun.com/pypi/simple/
# If you meet an "Undefined symbol" error while using VLLM_USE_PRECOMPILED=1, please use "pip install -e . -v" to build from source.
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e . --index-url https://mirrors.aliyun.com/pypi/simple/
For more details on compiling vLLM from source, refer to the vLLM official documentation.
Infer single wav file
# stage1: download pretrained model
# stage2: inference
source your_env_path/vllm_omni/bin/activate # source the env
bash ./inference_single_wav.sh
🙏 Acknowledgements
Special thanks to the following open-source projects:
Citation
If you find this work useful, please cite:
@misc{dai2026soulxtranscriber,
title={SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription},
author={Yuhang Dai and Haopeng Lin and Zhennan Lin and Jiale Qian and Jun Wu and Hanke Xie and Hao Meng and Hanlin Wen and Chuang Ding and Shunshun Yin and Ming Tao and Lei Xie and Xinsheng Wang},
year={2026},
eprint={2606.02400},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2606.02400},
}
License
We use the Apache 2.0 License. Researchers and developers are free to use the codes and model weights of our SoulX-Transcriber. Check the license at LICENSE for more details.
Contact
- Issues: Please open a GitHub Issue for bug reports or suggestions.
- Email: yhdai@mail.nwpu.edu.cn, haopenglin@soulapp.cn, lxie@nwpu.edu.cn, wangxinsheng@soulapp.cn