Whisper Kalenjin — LoRA fine-tune of whisper-large-v3-turbo

Kalenjin (Nilotic, ~6.3M speakers in Kenya) ASR adapter on top of openai/whisper-large-v3-turbo. Trained with LoRA on all 41 scripted-speech audio shards from Anv-ke/Kalenjin.

This is the merged full model (adapter baked into base weights, ~1.6 GB).

Quickstart

The recommended inference recipe uses chunked long-form decoding plus beam=5 plus hallucination-mitigation decode args. Beam=5 alone reduces WER by ~17 points compared to greedy; the rest of the args handle long-audio chunking and suppress the classic Whisper repetition-tail failure mode.

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="Tonykip/whisper-kalenjin-v3-turbo",
    chunk_length_s=30,
    stride_length_s=(5, 5),
)

result = pipe(
    "your_audio.wav",
    return_timestamps=True,
    generate_kwargs={
        "language": "sw",
        "task": "transcribe",
        "num_beams": 5,
        "compression_ratio_threshold": 1.35,
        "no_repeat_ngram_size": 3,
        "repetition_penalty": 1.15,
        "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
    },
)
print(result["text"])

Results (on 198-clip held-out unscripted dev_test set)

All numbers from canonical_metrics.json in the project repo. Recipe: chunked + beam=5 + temperature fallback + hallucination guards. Normalization: lowercase, [cs]/[pause] stripped, punctuation collapsed.

For comparison: greedy decoding (single best token per step) on the same model and same clips gives WER 82.79% / CER 33.88%. Beam search is the single biggest decoder improvement we found — bigger than any LoRA-recipe change.

Dialect-stratified at the recommended recipe, using ground-truth labels from Anv-ke/Kalenjin/dev_test/unscripted/files/meta.csv:

This model serves Kipsigis and Nandi speakers comparably — dialect gap of −0.57 WER points (Kipsigis very slightly better), within sampling noise on n=42 Nandi clips.

For comparison, base whisper-large-v3-turbo zero-shot WER on Kalenjin is 124% (measured on 200 scripted clips; the model hallucinates Icelandic-looking text and repetition loops on Kalenjin audio it has never been trained for). On English LibriSpeech, this fine-tune is roughly +5 WER points worse than base Whisper (almost entirely punctuation drift; words are correct).

Why CER, not just WER

For agglutinative languages with inconsistent orthographic conventions (apostrophe in ng', double letters, word-boundary ambiguity), WER counts whole-word mistakes that a native reader would call cosmetic. CER (character-level) correlates ~5 points better with human judgment across African ASR benchmarks. Both are reported.

Training

• Base: openai/whisper-large-v3-turbo (MIT)
• Method: LoRA, r=16, alpha=32, targets q_proj, k_proj, v_proj, out_proj, fc1, fc2
• Optimizer: AdamW, lr=1e-4, 50-step warmup, bf16 mixed precision
• Epochs: 2
• Hardware: 1× A100-40GB on Modal
• Data: 41 scripted-speech audio shards (~29 GB, the entire train/scripted/ split of Anv-ke/Kalenjin). Language token anchored to Swahili (sw) during training and inference.

Limitations and intended use

• Speech-style scope. Trained on scripted (read) speech. Spontaneous unscripted speech transcribes meaningfully worse (~66% normalized WER, ~21% CER on the eval set at the recommended recipe). Don't expect transcription quality on casual conversational audio to match dictation.
• Sub-tribe scope. "Kalenjin" is an umbrella term covering ~9 sub-tribes (Kipsigis, Nandi, Tugen, Keiyo, Marakwet, Sabaot, Pokot, Terik, Sengwer). The Anv-ke/Kalenjin dataset uses only KIPSIGIS and NANDI as dialect labels. Speakers from Keiyo, Marakwet, and Tugen counties are likely present in training data but rolled into one of the two main labels. Pokot, Sabaot, Terik, and Sengwer are likely absent from training data; speakers from those communities should expect noticeably worse accuracy than this model card reports.
• Long-form audio. Whisper's audio encoder has a 30-second receptive field. The recommended recipe handles long audio via chunked inference; do not pass long waveforms without chunking, or you'll get a transcription of only the first 30 seconds.
• Beam search vs latency. The 17-WER-point gain from beam=5 is not free — beam search is ~5× slower per chunk than greedy. For latency-critical applications (live dictation), greedy may be necessary; for batch transcription, use beam=5.
• English regression. ~5 WER points worse on English than base Whisper, mostly punctuation. If your application requires high-fidelity bilingual ASR, run base Whisper for English audio.
• Research artifact, not production. No SLA, no warranty. Don't deploy for safety-critical speech tasks (medical, legal, emergency response) without further evaluation on your specific audio domain.

Citation / attribution

If you use this model, please cite the underlying data source:

Wanzare, L., Amol, C., Maina, E., Odhiambo, N., Kerubo, H., Misula, L.,
Oloo, V., Mboya, R., Onkoba, E., Ombui, E., Muguro, J., wa Maina, C.,
Kipkebut, A., Otom, A.O., Kang'ethe, I.N., Kanyi, A.W., Omwenga, B.G.
(2025). AfriVoices-KE: A Multilingual Speech Dataset for Kenyan Languages.
arXiv:2604.08448. https://huggingface.co/datasets/Anv-ke/Kalenjin

openai/whisper-large-v3-turbo — https://huggingface.co/openai/whisper-large-v3-turbo

License

MIT — matches the base model. Training data licensing is governed by the Anv-ke/Kalenjin terms (CC BY 4.0 with gated access). Users intending to retrain or redistribute training data should request access from the dataset authors directly. This repo ships only model artifacts; no audio or transcripts are redistributed.