Wav2Vec2-BERT 2.0 ASR for 6 East African Languages

A fine-tuned version of the 580M-parameter, Conformer-based W2v-BERT 2.0 speech encoder with a character-level CTC head over a shared 34-character union vocabulary (Kikuyu ĩ/ũ preserved). One model transcribes all six languages — Dholuo (luo), Kalenjin (kln), Kikuyu (kik), Maasai (mas), Somali (som), Swahili (swa) — with no language token or language input: the model transcribes whichever of the six languages it hears.

Built for the AfriVoices East Africa ASR Hackathon (Kaggle; Digital Umuganda & Maseno Center for Applied Artificial Intelligence), where it has achieved a provisional 2nd place with 0.34819 macro-WER across the six languages. The hackathon corpus comprises scripted (read) and unscripted (spontaneous) speech across multiple dialects and regions, spanning Health, Government, Financial Services, Education, and Agriculture.

Output is lowercase without punctuation (neither capitalisation nor punctuation is modelled). Input: 16 kHz mono audio.

Usage

import torch
from transformers import Wav2Vec2BertProcessor, Wav2Vec2BertForCTC, pipeline

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
model_id = "leophill/w2v-bert-2.0-afrivoices-ea-6l-asr"

model = Wav2Vec2BertForCTC.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True,
    use_safetensors=True).to(device)
processor = Wav2Vec2BertProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch_dtype,
    device=device,
    chunk_length_s=28,       # long-form: windowing with overlap, as used in
    stride_length_s=4,       # the submission (28 s / 4 s)
)

audio_file = "audio.wav"
print(pipe(audio_file)["text"])

This is a CTC model: the pipeline call takes no generate_kwargs, no task, and no language argument. For best accuracy, decode the logits with pyctcdecode + a per-language 5-gram KenLM (beam 100; the submission used per-language α ∈ {0.25, 0.5}, β = 0.0; Swahili greedy).

Results

Kaggle leaderboard (test): 0.34819 macro-WER, 2nd place (winner's score: 0.34778). Development-set greedy WER (no language model), per language:

Test-set WER is register-sensitive: unscripted long-form speech is substantially harder than scripted read speech in every language.

Post-deadline inference recommendation

Decode whole clips — don't window. A post-deadline single-variable measurement on the official test set, decoding every clip in one pass (up to 101s) instead of 28s / 4s windows, improved macro-WER 0.34819 → 0.33930 (−0.0089) with the identical checkpoint and LM configuration. Per-window KenLM-state resets and overlap stitching are what windowing costs. The encoder extrapolates to ~3.4× its 30 s training length with no observed degradation, so windowing is only needed when memory forces it.

# ... same as above ...

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch_dtype,
    device=device
)

audio_file = "audio.wav"
print(pipe(audio_file)["text"])

Training

Fine-tuned jointly on all six languages (temperature-based sampling, α = 0.3) on the hackathon corpora plus rules-compliant public external audio for Dholuo and Kikuyu — Common Voice 26.0 (CC0), OpenBible audio, Thiomi-5k (CC-BY), WaxalNLP (CC-BY-SA) — quality-gated by a two-instrument agreement filter (forced alignment + model CER) with per-speaker caps. The released checkpoint is a low-learning-rate continued fine-tune of the best checkpoint-averaged joint model on the extended mix.

Edge version for mobile phones and CPU devices

Int8 conversions (CTranslate2, ONNX) run the full test set on a 4-core / 8 GB CPU box — see the edge model page.

Intended use & limitations

Speech recognition in the six covered languages only; not a language identifier (audio in other languages produces unusable text). Accuracy follows the training corpora's orthographic conventions and varies by register and dialect. 16 kHz mono input.

Citation

@misc{w2v-bert-2.0-afrivoices-ea-6l-asr,
  author = {Leopold Hillah},
  title = {Jointly Finetuning Wav2Vec-BERT 2.0 for 6 East African Languages},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/leophill/w2v-bert-2.0-afrivoices-ea-6l-asr}
}
@misc{afri-voices-east-africa-asr-hackathon,
  author = {{Digital Umuganda} and {Maseno University} and
            {Maseno Center for Applied Artificial Intelligence}},
  title = {AfriVoices East Africa: ASR Hackathon},
  year = {2026},
  howpublished = {\url{https://kaggle.com/competitions/afri-voices-east-africa-asr-hackathon}},
  note = {Kaggle}
}