A version with noise detection is trained base on this model, to reduce hallucination during streaming:
Name: JackyHoCL/whisper-large-v3-turbo-cantonese-noise-detection https://huggingface.co/JackyHoCL/whisper-large-v3-turbo-cantonese-noise-detection transformers-4.49.0 For Cantonese + English, use 'yue', for Cantonese + Mandarin + English, use 'zh'
TODO: 1.Improve zh-CN performance 2.Improve overall performance (yue+zh+en) with background noise (Please kindly suggest/provide dataset if possible, thx)
temperature=0.3, extra_body=dict( seed=4419, repetition_penalty=1.05, top_p=0.5 )
2025-07-21: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 8.05 |
| mozilla-foundation/common_voice_17_0 | yue | test | 0.64 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 8.3 |
| mozilla-foundation/common_voice_17_0 | en | test(2k samples) | 5.22 |
| mozilla-foundation/common_voice_16_1 | zh-CN | test | 11.89 |
2025-07-19: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 8.94 |
| mozilla-foundation/common_voice_17_0 | yue | test | 1.29 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 8.00 |
| mozilla-foundation/common_voice_17_0 | en | test | 6.8 |
| mozilla-foundation/common_voice_16_1 | zh-CN | test | 50.9 |
2025-07-06: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 8.92 |
| mozilla-foundation/common_voice_17_0 | yue | test | 8.86 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 7.96 |
| mozilla-foundation/common_voice_17_0 | en | test | 6.84 |
| mozilla-foundation/common_voice_16_1 | zh-CN | test | 43.0 |
per_device_train_batch_size=32, learning_rate=1e-7,
2025-07-03: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 9.705 |
| mozilla-foundation/common_voice_17_0 | yue | test | 9.31 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 8.37 |
per_device_train_batch_size=32, learning_rate=1e-5,
CER: 13.7%
Train Args: per_device_train_batch_size=16, gradient_accumulation_steps=1, learning_rate=1e-5, gradient_checkpointing=True, per_device_eval_batch_size=16, generation_max_length=225,
Hardware: NVIDIA Tesla V100 16GB * 4
A Realtime Streaming application example is built on this model: https://github.com/JackyHoCL/whisper-realtime.git
FAQ:
- If having tokenizer issue during inference, please update your transformers version to >= 4.49.0
pip install --upgrade transformers