Kinyarwanda Whisper Evaluation

This repository evaluates the Whisper model performance on Kinyarwanda, as described in the paper How much speech data is necessary for ASR in African languages? An evaluation of data scaling in Kinyarwanda and Kikuyu.

Model Description

The development of Automatic Speech Recognition (ASR) systems for low-resource African languages remains challenging due to limited transcribed speech data. This work addresses fundamental concerns for practitioners, evaluating Whisper's performance through comprehensive experiments on Kinyarwanda. Systematic data scaling analysis on Kinyarwanda, using training sets from 1 to 1,400 hours, demonstrated that practical ASR performance (WER < 13%) becomes achievable with as little as 50 hours of training data, with substantial improvements continuing through 200 hours (WER < 10%).

For more details on the evaluation, training, and related models, visit the GitHub repository.

Training Configs

The following models were used in the Kinyarwanda Whisper evaluation, trained with different data volumes. Explore the full collection: 👉 https://huggingface.co/collections/Sunbird/kinyarwanda-hackathon-68872541c41c5d166d9bffad

Evaluation Results

Evaluation on dev_test[:300] subset:

Score = 1 - (0.6 × CER + 0.4 × WER)