On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin
This repository hosts the released CN-MultiDialect-ASR checkpoint, adapted from Qwen3-ASR-1.7B with a three-stage pipeline: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and On-Policy Self-Distillation (OPSD). The goal is to improve Chinese dialect recognition without raising Mandarin CER.
- Paper: arXiv:2608.11898
- Code, demo, and training scripts: ASLP-lab/CN-MultiDialect-ASR
Demo
Video demo with live waveforms and model transcriptions for Mandarin, English, four core dialects, and 15 ChinaVoices dialects.
Key Features
- Mandarin–dialect balanced adaptation: improves Chinese dialect ASR while retaining Mandarin recognition.
- Three-stage pipeline: CPT strengthens the Chinese ASR foundation, dialect SFT specializes for dialects, and OPSD refines the final checkpoint.
- On-Policy Self-Distillation: trains on student-decoded prefixes with soft teacher targets, reducing the train–test mismatch of teacher-forced ASR training.
- Drop-in inference: compatible with the official
qwen-asrpackage.
Quickstart
Inference is compatible with Qwen3-ASR. We recommend installing the official qwen-asr package in a clean environment.
Environment Setup
conda create -n qwen3-asr python=3.12 -y
conda activate qwen3-asr
pip install -U qwen-asr
For faster inference with the vLLM backend:
pip install -U qwen-asr[vllm]
Model Download
You can load the model directly from Hugging Face, or download it locally first:
# Hugging Face
pip install -U "huggingface_hub[cli]"
hf download ASLP-lab/CN-MultiDialect-ASR --local-dir ./CN-MultiDialect-ASR
# ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model ASLP-lab/CN-MultiDialect-ASR --local_dir ./CN-MultiDialect-ASR
Python Inference
Load the model with Qwen3ASRModel.from_pretrained and call transcribe:
import torch
from qwen_asr import Qwen3ASRModel
model = Qwen3ASRModel.from_pretrained(
"ASLP-lab/CN-MultiDialect-ASR", # or "./CN-MultiDialect-ASR" for a local path
dtype=torch.bfloat16,
device_map="cuda:0",
# attn_implementation="flash_attention_2",
max_inference_batch_size=32,
max_new_tokens=256,
)
results = model.transcribe(
audio="path/to/audio.wav",
language="Chinese", # or None for automatic language detection
)
print(results[0].language)
print(results[0].text)
Batch inference is also supported:
results = model.transcribe(
audio=[
"path/to/mandarin.wav",
"path/to/dialect.wav",
],
language=["Chinese", "Chinese"],
)
for r in results:
print(r.language, r.text)
For vLLM backend, streaming inference, and forced alignment, see the Qwen3-ASR repository.
Method Overview
| Stage | Training data | Goal | Objective |
|---|---|---|---|
CPT | Full Mandarin-dialect collection (~100k hours) | Build a stronger Chinese ASR foundation | Cross-entropy |
SFT | Same sources with higher dialect sampling weight and a small Mandarin anchor | Lower dialect CER | Cross-entropy |
OPSD | Dialect refinement subset (~5k hours) | Improve dialect recognition without hurting Mandarin | Token-level KL |
At inference time, only the student pathway is used.
Performances
Dialect Overview
Public Dialect CER (%)
Internal Dialect CER (%)
Mandarin CER (%)
Citation
If you use this model, please cite:
@misc{wang2026onpolicyselfdistillationmultidialectasr,
title={On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin},
author={Shuiyuan Wang and Bingshen Mu and Pengshen Zhang and Chengyou Wang and Yujie Liao and Chengdong Liang and Binbin Zhang and Qiangze Feng and Lei Xie},
year={2026},
eprint={2608.11898},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2608.11898}
}
License
The released model is licensed under Apache 2.0.
Contact
For questions or collaborations, please contact wangshuiyuan@mail.nwpu.edu.cn.
You are also welcome to join our WeChat group for technical discussions and updates.