Yaw ASR — joint Twi + English (300M CTC)

Author: Prince Nasamu Alhassan · Project: Yaw — an ambient clinical scribe for Ghanaian outpatient consultations (Twi and English, often code-switched). AI proposes, doctor decides.

Current release of the Yaw ASR campaign (v13d, October 2026). A single Wav2Vec2ForCTC (315.5M params) that transcribes Twi (Akan), Ghanaian English, and Twi–English code-switching with one character head. Superseded runs are kept as separate repos (-v13b, -v13c); earlier intermediates (v4–v11) were deleted.

Results

val, test_b and test_w are speaker-disjoint AND text-disjoint from training: a load-time guard refuses any training row whose normalised text appears in one of them. test_k is the exception and is text-contaminated — see the external benchmark section below. For a benchmark with no overlap of either kind, use the nsanku-ASR results.

Code-switching — the register this model exists for — is scored on the code-switched slice of val: 0.2144 WER, from 0.2408 in the previous release.

What changed in this release (October 2026). Training added 60 hours of real, manually transcribed English–Twi code-switching from the KasaSpeech corpus (the 6 speakers that appear in test_b were excluded by speaker id), replacing an earlier self-training slice. Code-switching, English and the external benchmark all improved; test_w regressed by 0.5 points (0.3594 → 0.3644) and test_b by 0.1, because WAXAL image-description speech is now a smaller share of a much larger Twi mixture. That trade is deliberate: code-switched consultation speech is the target register, image description is not. The previous release remains available as yaw-asr-twi-en-v13b and the pre-KasaSpeech model in the training repos.

KasaSpeech test split — new speakers, SEEN SENTENCES. Read the caveat. On the full 1,725 clips (4.8 h, 40,766 reference words) of the official KasaSpeech test split, every speaker disjoint from this model's training:

Measured with the yaw-kasa-audit harness on 2026-10-05 and 2026-10-07.

This number is weaker than it looks, and it used to be billed on this card as the most trustworthy one. It is not. KasaSpeech's test split contains exactly 803 distinct normalised transcripts, and all 803 also occur in its train split — the split this model trained 60.1 h of. The trainer's text guard filters against our val/test_b/test_w, never against their test; in a 2,000-row sample of their train split, 65.9% of rows carried a sentence that is also a test sentence. So 17.3% measures generalisation to new voices on sentences seen in training in other voices. It is a real result — speaker and channel generalisation on a tonal low-resource language is not easy — but it is not open-vocabulary generalisation, and it should not be compared against anyone's text-disjoint score.

The same corpus structure explains the LM result below: 647 of those 803 test sentences are in the 9,022-sentence corpus the 5-gram was built from.

For a code-switching number with no text overlap, use test_b's KasaSpeech slice, 0.2980 (speaker- and text-disjoint, the 6 overlapping speakers excluded by id).

External benchmark, text-disjoint: GhanaNLP nsanku-ASR. Scored on ghana-speech-eval with GhanaNLP's own nsanku-asr-benchmark metric code copied verbatim (per-sample WER/CER macro-averaged within a category, then categories averaged with equal weight; their leaderboard ranks on CER). Overlap against this model's training text was measured before running anything:

Both finance categories are the Ashesi FISD corpus, 0.7 h of which is in this model's training mixture, so they are reported but never averaged into the headline.

Per clean category:

Asante Twi, clean categories only (3,196 clips): WER 0.5822 / CER 0.2913 — which places this model 3rd of 19 on that leaderboard recomputed over the same four categories, behind Qlerqly/griot-nano-1 (0.2607 CER) and Qlerqly/griot-edge (0.2624), ahead of KhayaAI, Google, Sunbird, FarmerlineML, both Gemini models and facebook/omniASR-LLM-7B-v2.

Akuapem Twi: WER 0.6813 / CER 0.3273, from bible alone — the only clean Akuapem category — so it rests on one register and is not a language-level score. On it this model is level with the checkpoint it started from, which is the honest reading: the training mixture carries 60.1 h of Asante-leaning code-switched conversation against 2.9 h of read Akuapem, and no scripture audio at all.

That register gap is the main story in the per-category spread. This model is strongest on waxal image-description narration (0.3080, better than griot-nano-1's 0.3461) and weakest on scripture, where models built on BibleTTS-style corpora dominate — the inverse of most open Twi ASR, and the right trade for a clinical scribe. On waxal + unicef together, the two registers closest to spontaneous speech, this model is 0.518/0.256 against griot-nano-1's 0.525/0.254 and griot-edge's 0.456/0.259.

What including the contaminated categories would do, for transparency: the leaderboard's own definition averages finance in, giving Asante 0.5058/0.2471 (CER 15.2% better than clean) and Akuapem 0.4395/0.2073 (36.7% better, because finance is half of Akuapem's categories). Asante stays 3rd either way; Akuapem would move from 4th to 2nd. Both the clean and board figures are emitted by the harness and neither is collapsed into the other.

Harness: asr/kaggle/yaw-nsanku-bench/ in the training repo, with the overlap measurement in overlap_vs_yaw_training.json. Note this model is published under a personal account, so it is not eligible for that leaderboard, which lists org-owned models only; these numbers are self-reported using their code and data.

The folded column strips ɛ/ɔ and combining marks for comparison with benchmarks that fold diacritics; the primary score preserves them (they are phonemic in Twi).

Safety slices. Hallucination false-trigger rate: 0% on pure silence, 0% on very short clips, 0% on speech with 3 s injected pauses (20 clips each) — the model does not invent words over silence. Number-bearing utterances (NWER): 0.29 on test_w (n=154), 0.28 on test_en (n=84). No eval set contains medication mentions, so drug-slice WER is unmeasurable on this data — clinical-domain evaluation uses PriMock57 separately and Ghanaian clinical Twi collection is the open need.

Per-source detail (WER):

* the val Asante slice is 10 minutes of audio from very few speakers; the test_b figure (0.4043) is the meaningful one. Asante Twi is the weakest slice in every run — it is 9.8 minutes of the training mixture.

English detail (test_en WER): AMI close-mic meetings 0.3628, AMI single-distant-mic 0.5678, SLUE call-centre conversations 0.0864. Distant-microphone meeting audio is the known weak slice; close-mic conversational English is strong.

Full numbers, including the exact eval pairs, are in eval_yaw101.json and eval_pairs.json in this repo.

How it was trained

1. Warm start: uctnlp/omniASR-CTC-300M-v2-Zulu-Baseline — a supervised multilingual CTC checkpoint (10,288-class character head) whose training already included 45.1 h of Akan.
2. Head pruning: the 10,288-class head was sliced to the 71 classes this task can emit (a–z, Akan ɛ/ɔ and diacritics, digits, punctuation, space as the word boundary, CTC blank <s>). Slicing preserves the pretrained weights row-for-row and was verified by proving greedy output identical to the full head with non-kept logits masked to −inf.
3. Main run: 14-epoch schedule at lr 1e-4 (effective batch 32) on 64.6 h; stopped by the session wall at epoch 8.35 (val WER 0.3414).
4. Continuation: fresh optimizer at lr 2e-5 (warmup 0.05) for 6 further epochs from the epoch-8.35 weights. Model selection by WER, not eval loss (CTC eval loss rises while WER keeps improving — peaky posteriors). Best epoch: val WER 0.2963.
5. Self-training (superseded): ~900 h of unlabelled WAXAL Akan was pseudo-labelled by the step-4 model and gated by CROSS-MODEL agreement with an unrelated w2v-BERT CTC Twi model (96.6% keep rate); the strictest 80 h slice trained 3 further epochs and every eval set improved slightly. A second round on a disjoint slice did NOT: it gained 0.7 points but began inventing words on short clips, and was rejected. No pseudo-labelled audio is in this release.
6. Real code-switching (this release): that 80 h slice was replaced by 60.1 h of manually transcribed English–Twi code-switching from the official KasaSpeech train split. The 6 KasaSpeech speakers that appear in test_b were excluded by speaker id and the held-out-text guard still applies, so the external KasaSpeech test remains genuinely unseen. Mixture 126.9 h, 3 epochs at lr 2e-5.
7. Non-speech negatives: 60 h of long, prompt-read audio taught the model that audio always implies words — the false-trigger rate on 0.6 s near-silent clips went from 0% to 30%. Adding 1.8% synthetic non-speech rows with EMPTY labels (digital silence, noise floor, room tone) restored 0% across all three hallucination slices. CTC represents silence natively as the all-blank path; the mixture simply never contained an example of it.
8. Final pass: one epoch at lr 1e-5 from the best-accuracy checkpoint, with the negatives — keeping the accuracy while buying the silence behaviour. These are the published weights.
9. LM rescoring (negative result, reproduced 4×): a 5-gram KenLM over the available ~9k Twi sentences HURTS this model at every swept alpha/beta (best val 0.4077 vs 0.2723 greedy). It appeared to help test_k (0.2133 → 0.1085), which is contamination, not decoding: 647 of that test split's 803 distinct sentences are in the 9,022-sentence corpus the LM was built from. Decoding is greedy; no LM until a real Twi text corpus exists.

Training data (126.9 h, all human-transcribed)

Language balance: 51,106 Twi rows / 10,107 English rows.

Label-provenance rule: only human-transcribed corpora train the model. The large machine-labelled Twi corpora that exist (Google-STT / Gemini transcripts) were evaluated and excluded — label quality beats quantity at this scale.

Use it

import torch, soundfile as sf
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC

repo = "PrinceAlhassanNasamu/yaw-asr-twi-en"
proc = Wav2Vec2Processor.from_pretrained(repo)
model = Wav2Vec2ForCTC.from_pretrained(repo).eval()

wav, sr = sf.read("clip.wav", dtype="float32")   # 16 kHz mono, up to ~30 s
inp = proc(wav, sampling_rate=16_000, return_tensors="pt")
with torch.no_grad():
    ids = model(**inp).logits.argmax(-1)
print(proc.batch_decode(ids)[0])

Notes for integrators:

• Clip length: trained on 0.5–30 s clips. Segment longer audio (VAD) before decoding; the Yaw pipeline uses Silero VAD with a 28 s ceiling.
• Word timing: CTC frame stride is 320 samples (20 ms at 16 kHz); per-character frame indices give word timestamps and the per-word confidence is the mean emitted posterior. No forced aligner needed.
• Tokenizer quirk: the word boundary class is the literal space character (vocab id 0 is the blank <s>). The reloaded word_delimiter_token shows |; decoding is unaffected because space tokens pass through literally.
• preprocessor_config.json sets return_attention_mask=True — required for this architecture family; always pass the attention mask for batched inference.

Limitations

• Trained overwhelmingly on read/prompted speech. Real consultation audio (spontaneous, overlapping, far-field) is harder; PriMock57 clinical English WER in earlier campaign measurements was ~0.44 zero-shot. Ghanaian clinical audio collection is the planned fix.
• Distant-microphone audio is the weakest English slice (AMI SDM 0.58).
• Fante and other Akan varieties beyond Asante/Akuapem Twi are barely represented.
• Numerals in Ghanaian clinical speech are a measured ASR weak point; downstream systems should cross-check dosages against the audio (Yaw's pipeline does).

Licence and attribution

Released CC-BY-NC-4.0 (non-commercial): the training mix includes CC-BY-NC corpora (twi-health-500h, Ghana-2700, AfriSpeech-200, UGSpeechData and others), so this model and its derivatives are for research and non-commercial use, with attribution to the upstream corpus creators: WAXAL (Google), KasaSpeech, Ashesi FISD, GhanaNLP/ghanaopenai, Intron Health (AfriSpeech), University of Ghana (UGSpeechData), Babylon Health (PriMock57), AMI Consortium, ASAPP (SLUE), and UCT NLP (omniASR warm start).