[!WARNING] This is a preview release. The model weights will change between now and the final release. Benchmark numbers, prompt behaviour and output quality may all shift. Please don't pin production systems to this checkpoint — treat it as a preview for evaluation and feedback.

Whisper large for 51 African languages

This model is an adaptation of Whisper large-v3 that supports Automatic Speech Recognition (ASR) for 51 languages widely spoken in Africa: English and French with African accents, Swahili, Afrikaans, Tswana, Kinyarwanda, Nigerian Pidgin, Luganda, Acholi, Lugbara, Ateso, Runyankole, Rutooro, Lumasaba, Lusoga, Rukiga, Akan, Amharic, Bambara, Bemba, Berber, Chichewa, Dagaare, Dagbani, Ewe, Fulani, Hausa, Igbo, Ikposo, Kabyle, Kalenjin, Kanuri, Kikuyu, Kwamba, Lendu, Lingala, Luhya, Luo, Malagasy, Ndebele, Oromo, Rukonjo, Ruruuli, Shona, Somali, Sotho, Thur, Wolof, Xhosa, Yoruba, and Zulu.

We compared our model's ASR performance against other state-of-the-art proprietary and open-source models: Gemini 3.5 Flash, GPT-4o Transcribe, and Meta's Omnilingual ASR (omniASR). Our model achieves lower Word Error Rate (WER) and Character Error Rate (CER) on most African languages. See the details in the Performance Metrics section below.

Usage

You can use this Colab Notebook to try out the model. It lets you transcribe an audio sample from a Hugging Face dataset, an uploaded audio file, or your own recording from your computer's microphone.

The model is used in much the same way as the base Whisper model. For better accuracy, specify the language during generation.

See the example below, which transcribes an audio sample from a Hugging Face dataset:

import transformers
import datasets
import torch

SAMPLE_RATE = 16000

LANGUAGE_TOKENS_WHISPER = {
    # Existing languges codes from Whisper
    "eng": 50259, "fra": 50265, "swa": 50318, "sna": 50324, "yor": 50325, "som": 50326,
    "afr": 50327, "amh": 50334, "mlg": 50349, "lin": 50353, "hau": 50354,
    # Overwrite unused language tokens
    "ach": 50357, "aka": 50356, "bam": 50355, "bem": 50352, "ber": 50351,
    "cgg": 50350, "dag": 50348, "dga": 50347, "ewe": 50346, "ful": 50345,
    "ibo": 50344, "kab": 50343, "kau": 50342, "kik": 50341, "kin": 50340,
    "kln": 50339, "koo": 50338, "kpo": 50337, "led": 50336, "lgg": 50335,
    "lth": 50333, "lug": 50332, "luo": 50331, "luy": 50330, "myx": 50329,
    "nbl": 50328, "nya": 50323, "nyn": 50322, "orm": 50321, "pcm": 50320,
    "ruc": 50319, "rwm": 50317, "sot": 50316, "teo": 50315, "tsn": 50314,
    "ttj": 50313, "wol": 50312, "xho": 50311, "xog": 50310, "zul": 50309
}

LANGUAGE_NAMES = {
    'Acholi': 'ach', 'Afrikaans': 'afr', 'Akan': 'aka', 'Amharic': 'amh', 'Ateso': 'teo',
    'Bambara': 'bam', 'Bemba': 'bem', 'Berber': 'ber', 'Chichewa': 'nya', 'Dagaare': 'dga',
    'Dagbani': 'dag', 'English': 'eng', 'Ewe': 'ewe', 'French': 'fra', 'Fulani': 'ful',
    'Hausa': 'hau', 'Igbo': 'ibo', 'Ikposo': 'kpo', 'Kabyle': 'kab', 'Kalenjin': 'kln',
    'Kanuri': 'kau', 'Kikuyu': 'kik', 'Kinyarwanda': 'kin', 'Kwamba': 'rwm', 'Lendu': 'led',
    'Lingala': 'lin', 'Luganda': 'lug', 'Lugbara': 'lgg', 'Luhya': 'luy', 'Lumasaba': 'myx',
    'Luo': 'luo', 'Lusoga': 'xog', 'Malagasy': 'mlg', 'Ndebele': 'nbl', 'Nigerian Pidgin': 'pcm',
    'Oromo': 'orm', 'Rukiga': 'cgg', 'Rukonjo': 'koo', 'Runyankole': 'nyn', 'Ruruuli': 'ruc',
    'Rutooro': 'ttj', 'Shona': 'sna', 'Somali': 'som', 'Sotho': 'sot', 'Swahili': 'swa',
    'Thur': 'lth', 'Tswana': 'tsn', 'Wolof': 'wol', 'Xhosa': 'xho', 'Yoruba': 'yor', 'Zulu': 'zul'
}

model_id = "Sunbird/asr-whisper-51-african-languages"
model = transformers.WhisperForConditionalGeneration.from_pretrained(model_id)
processor = transformers.WhisperProcessor.from_pretrained(model_id)

def transcribe_by_whisper(audio_array, language):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    input_features = processor(
        audio_array, sampling_rate=SAMPLE_RATE, do_normalize=True, return_tensors="pt"
    ).input_features.to(device)

    lang_tok = LANGUAGE_TOKENS_WHISPER[LANGUAGE_NAMES[language]]
    transcribe_tok = processor.tokenizer.convert_tokens_to_ids("<|transcribe|>")
    notimestamps_tok = processor.tokenizer.convert_tokens_to_ids("<|notimestamps|>")
    forced_decoder_ids = [
        (1, lang_tok),
        (2, transcribe_tok),
        (3, notimestamps_tok),
    ]

    predicted_ids = model.to(device).generate(
        input_features,
        forced_decoder_ids=forced_decoder_ids,
        num_beams=1,
        do_sample=False,
    )
    transcription = processor.decode(
        predicted_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)

    print(transcription[0])

# Get some test audio
import huggingface_hub
huggingface_hub.login()

ds = datasets.load_dataset('Sunbird/salt', 'multispeaker-lug', split='test')
audio_column = 'audio'
ds = ds.cast_column(audio_column, datasets.Audio(sampling_rate=SAMPLE_RATE))
audio_array = ds[0][audio_column]['array']

# Specify a language
lang = 'Luganda'

transcribe_by_whisper(audio_array, lang)
# Ekikoola kya kasooli kya kyenvu wabula langi yakyo etera okuba eya kitaka wansi.

Performance Metrics

In the chart below, we compare our model's ASR performance against other state-of-the-art proprietary and open-source models: Gemini 3.5 Flash from Google, GPT 4o Transcribe from OpenAI, Omnilingual ASR (OmniASR) from Meta. Our model exceeds the performance of these models, especially on low-resource languages such as Acholi and Ateso from Uganda, while achieving comparable performance on many widely spoken African languages such as Swahili, as well as on English and French spoken with African accents.

Word Error Rate (WER) and Character Error Rate (CER) are measured for each language on the evaluation datasets. For ach, eng, lgg, lug, nyn, and teo, the dev splits of the multispeaker-* subsets in the Sunbird/salt dataset were used for evaluation. For the remaining languages, we used the first 50 examples from the dev split of each subset in the Training Datasets described in the section below, after filtering out clips longer than 30 seconds (the Whisper input limit).

code   language            WER      CER    n_examples
afr    Afrikaans         0.084    0.028    170
aka    Akan              0.323    0.129     50
amh    Amharic           0.401    0.265    250
bam    Bambara           0.361    0.170    149
bem    Bemba             0.408    0.091    100
ber    Berber            0.147    0.041     50
cgg    Rukiga            0.242    0.057     49
dag    Dagbani           0.321    0.109    100
dga    Dagaare           0.294    0.113     50
ewe    Ewe               0.291    0.097     78
fra    French            0.032    0.019     50
ful    Fulani            0.395    0.115    149
hau    Hausa             0.168    0.047    287
ibo    Igbo              0.340    0.117    247
kab    Kabyle            0.447    0.219     50
kau    Kanuri            0.389    0.101     50
kik    Kikuyu            0.209    0.099     50
kin    Kinyarwanda       0.307    0.084    100
kln    Kalenjin          0.471    0.112    100
koo    Rukonjo           0.527    0.093     47
kpo    Ikposo            0.617    0.231     50
led    Lendu             0.273    0.093     50
lin    Lingala           0.243    0.104    193
lth    Thur              0.148    0.059     41
luo    Luo               0.226    0.051    149
luy    Luhya             0.203    0.038     50
mlg    Malagasy          0.143    0.060     49
nbl    Ndebele           0.263    0.051     10
nya    Chichewa          0.273    0.058     98
orm    Oromo             0.241    0.064    119
pcm    Nigerian Pidgin   0.167    0.065    100
ruc    Ruruuli           0.480    0.105     38
rwm    Kwamba            0.584    0.234     49
sna    Shona             0.196    0.050    147
som    Somali            0.381    0.127    149
sot    Sotho             0.229    0.117     94
swa    Swahili           0.087    0.022     99
tsn    Tswana            0.067    0.018     98
ttj    Rutooro           0.222    0.063     49
wol    Wolof             0.319    0.122    135
xho    Xhosa             0.262    0.054    126
yor    Yoruba            0.384    0.136    249
zul    Zulu              0.207    0.042    143
ach    Acholi            0.197    0.044    101
eng    English           0.041    0.016    101
lgg    Lugbara           0.216    0.052    101
lug    Luganda           0.109    0.022    103
nyn    Runyankole        0.262    0.051    103
teo    Ateso             0.275    0.068    101
myx    Lumasaba          0.347    0.071    100
xog    Lusoga            0.301    0.056    100

Training Datasets

The model was fine-tuned on a multilingual corpus covering 51 African languages, assembled from a range of publicly available and community-collected speech datasets. The table below lists each source dataset and the languages it contributes to training (ISO 639-3 code in parentheses). A language may appear in more than one source.

Notes

• African Next Voices is drawn from two collection hubs: Kenya (Kikuyu, Kalenjin, Luo, Somali) and Southern Africa (Ndebele, Sotho, Tswana, Xhosa, Zulu).