Important Note:
This model is for streaming purpose, to reduce false positive results(hallucinations) in silent/background noise The noise is labeled as "%nz", please remove/replace this pattern in the transcribe result
temperature=0.3, extra_body=dict( seed=4419, repetition_penalty=1.05, top_p=0.5 ),
2025-10-07: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 6.19 |
| mozilla-foundation/common_voice_22_0 | yue | test | 1.77 |
| mozilla-foundation/common_voice_17_0 | yue | test | 0.751 |
| mozilla-foundation/common_voice_17_0 | en | test(2k samples) | 5.55 |
| mozilla-foundation/common_voice_16_1 | zh-CN | test | 8.02 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 7.68 |
| Sunbird/urban-noise-uganda-61k:small(1k) | noise | half(500) | 0 |
2025-07-21: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 3.234 |
| mozilla-foundation/common_voice_17_0 | yue | test | 0.437 |
| mozilla-foundation/common_voice_17_0 | en | test(2k samples) | 5.18 |
| mozilla-foundation/common_voice_16_1 | zh-CN | test | 11.74 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 9.86 |
| Sunbird/urban-noise-uganda-61k:small(1k) | noise | half(500) | 12.7 |
2025-07-06: CER:
| Dataset | Lang | Split | CER(in %) |
|---|---|---|---|
| Training | yue | validation | 8.95 |
| mozilla-foundation/common_voice_17_0 | yue | test | 8.78 |
| mozilla-foundation/common_voice_16_1 | yue | test | 8.76 |
| JackyHoCL/cleaned_mixed_cantonese_and_english_speech | yue | test | 8.00 |
| Sunbird/urban-noise-uganda-61k:small(1k) | noise | half(500) | 0.0 |
Train Args:
per_device_train_batch_size=32, learning_rate=1e-6, gradient_accumulation_steps=1, gradient_checkpointing=True, per_device_eval_batch_size=16, generation_max_length=225,
Hardware: NVIDIA Tesla V100 16GB * 4
A Realtime Streaming application example is built on this model: https://github.com/JackyHoCL/whisper-realtime.git
FAQ:
- If having tokenizer issue during inference, please update your transformers version to >= 4.46.3
pip install --upgrade transformers