Whisper Medium for Mandarin Singing Lyrics
whisper-medium-m4singer-zh-lyrics is a full-parameter fine-tune of
openai/whisper-medium for
Mandarin singing-voice transcription. It was trained on the M4Singer corpus
with song-disjoint train, validation, and test splits.
Intended use
The model transcribes short Mandarin singing phrases into Chinese lyrics. It is intended for non-commercial research and evaluation. It is not designed for speech translation, speaker identification, or unrestricted commercial use.
Data and license
Training used M4Singer under its CC BY-NC-SA 4.0 non-commercial terms:
- 20,896 accepted segments; 0 rejected
- 29.70 hours
- 20 singers and 419 songs
- 16,834 train / 1,988 validation / 2,074 test segments
- song, file-path, and audio-hash overlap across splits: zero
Users must comply with the M4Singer license and citation requirements. The training audio is not redistributed in this repository.
Training
- Base model:
openai/whisper-medium - Method: full-parameter fine-tuning
- Language/task:
zh/transcribe - Hardware: one NVIDIA RTX 6000 Ada Generation (48 GB)
- Precision: BF16; TF32 enabled
- Gradient checkpointing: enabled
- Per-device batch size: 2
- Gradient accumulation: 16
- Learning rate: 5e-6, linear schedule
- Epochs/steps: 4 / 2,108
- Runtime: 3 h 48 min
- Seed: 42
- Best checkpoint selected on validation: step 2,108
- PyTorch peak allocated memory: 6.23 GiB (not an
nvidia-smipeak)
Results
All CER values use the same configurable Chinese normalization and micro character aggregation.
| System | Validation CER |
|---|---|
| Original Whisper medium | 24.247% |
| This model | 22.354% |
This is an absolute improvement of 1.893 percentage points and a relative improvement of approximately 7.81%. The independent final test evaluation and song-level paired bootstrap analysis were queued when this initial model card was published; the card will be updated after those results are finalized.
Usage
import soundfile as sf
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
model_id = "Hengyuhan/whisper-medium-m4singer-zh-lyrics"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id, dtype=torch.float16, attn_implementation="sdpa"
).to("cuda")
audio, sample_rate = sf.read("singing.wav", dtype="float32")
inputs = processor(
audio, sampling_rate=sample_rate, return_tensors="pt",
return_attention_mask=True,
)
features = inputs.input_features.to("cuda", dtype=torch.float16)
mask = inputs.attention_mask.to("cuda")
model.generation_config.language = "zh"
model.generation_config.task = "transcribe"
with torch.inference_mode():
generated = model.generate(features, attention_mask=mask, max_new_tokens=440)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])
Audio should be mono and no longer than Whisper's approximately 30-second input window. Resample to 16 kHz when necessary.
Limitations
- M4Singer is primarily clean, professionally recorded singing; robustness to accompaniment, live recordings, reverberation, and consumer microphones is not established.
- Unseen-singer, external-dataset, and hallucination evaluations are not yet complete.
- The model can insert or repeat lyrics during silence, breaths, or instrumental passages. Do not treat generated text as verified ground truth.
- Performance is not evidence of broad Mandarin conversational ASR quality.
Citation
Please cite both Whisper and M4Singer when using this model. M4Singer:
@inproceedings{zhang2022msinger,
title={M4Singer: A Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus},
author={Zhang, Lichao and Li, Ruiqi and Wang, Shoutong and Deng, Liqun and Liu, Jinglin and Ren, Yi and He, Jinzheng and Huang, Rongjie and Zhu, Jieming and Chen, Xiao and Zhao, Zhou},
booktitle={NeurIPS Datasets and Benchmarks Track},
year={2022}
}