whisper-nl-noise

This model is a fine-tuned version of openai/whisper-medium on the the (dutch) mozilla/common-voice dataset (11.0). This dataset is augmented with various forms of background noise, retrieved from pixabay and outtakes from specific audio tracks.

Model description

Whisper (base)

Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Trained on 680k hours of labelled data, Whisper models demonstrate a strong ability to generalize to many datasets and domains without the need for fine-tuning.

Whisper was proposed in the paper Robust Speech Recognition via Large-Scale Weak Supervision by Alec Radford et al from OpenAI. The original code repository can be found here.

Whisper NL (noise)

This current Whisper model is specifically fine-tuned on noisy Dutch data. It intends to have increased performance on this task. The generalizing aspects of the model will however be lost in this process. It will nevertheless build upon the 680 hours of labeled data the base model already received during training.

Intended uses & limitations

This model is specifically trained for (very) noise (Dutch) audio. It is expected that is performs worse on audio files which do not meet these criteria.

Noise is often bound to specific contexts and recordings. The model will therefore not generalize to all sorts and types of (car) noise.

Training and evaluation data

The mozilla/common-voice dataset (11.0) was used. With the predefined 'train' and 'test' split. For reasons of time-management, only the first 5% of the test set was used.

Training procedure

The training procedure as outlined in the original Huggingface blog was used, see here.

The only alteration was made during preparation of the dataset (prepare_dataset(batch)). Within this step the data was augmented to include various samples of background noise. These noises were sampled from various audio files and sources. For each audio track used from the common-voice dataset, a random noise sample was mixed with this audio track. The signal to noise ratio (SNR) varied (randomly) between -5, and 1 dB.

$$ \mathrm{SNR} = {{P_{signal}} \over {P_{noise}}} $$

$$ \mathrm{SNR_{dB}} = 10 \log _{10}\mathrm{SNR} $$

This SNR range makes the noise disturbance, on average, quite invasive. The augmentation, does, however not alter the transcription of the audio track, these remain unchanged.

Whilst it would be beneficial to add the noise on the fly during training, to enable varying selections of noise for the same audio track, for efficiency reasons this strategy was not applied. Each audio tracks is therefore augmented with a single (randomly selected) noise track. The length of the audio track remained unchanged. If needed the noise track was repeated or truncated to meet the length of the audio track.

Training hyperparameters

- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- training_steps: 10000
- mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.37.2
• Pytorch 2.1.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2