TubaSTT — Twi+English Bilingual Conversational ASR, phone-robust (w2v-BERT, v0.5)
Bilingual, code-switch-aware, phone-robust speech recognition for Asante Twi / Akan
and English, for Ghanaian customer-service voice agents (8 kHz telephony). CTC fine-tune of
facebook/w2v-bert-2.0 (580M, MIT) on real
conversational + read Ghanaian speech + English, with phone-codec + SpecAugment augmentation.
v0.5 adds 8 kHz μ-law codec + SpecAugment augmentation over v0.4, cutting phone-codec WER substantially while slightly improving clean — the phone gap is nearly closed.
Results — WER by source (dev*)
| Source | Type | v0.4 WER | v0.5 WER | v0.5 +KenLM |
|---|---|---|---|---|
| BibleTTS unseen | Twi read | 9.8 | 9.1 | 8.0 |
| BibleTTS — 8 kHz phone | Twi read, telephony | 11.7 | 9.5 | — |
| WAXAL Akan | conversational + code-switch | 27.0 | 26.3 | 25.8 |
| WAXAL — 8 kHz phone | conversational, telephony | 33.0 | 27.9 | — |
| FLEURS ‡ | English | 11.7 | 11.7 | 6.4 ‡ |
| Ashesi FISD † | Twi read, fintech | 1.2 | 1.0 † | — |
* These are dev-set numbers. A speaker-disjoint, held-out test split is in construction and will be published alongside the next release. We report the caveats below rather than omit them, because two of these rows are not generalisation claims.
† Ashesi FISD is in-domain and should not be read as a headline. The corpus authors state that speakers and phrases in the 10% eval split may also appear in the 90% train split; we measured 100% transcript overlap and 100% speaker overlap with our training data. The 1.0 WER is reported for completeness and continuity with v0.4, not as evidence of generalisation.
‡ Treat the KenLM-fused English number as optimistic. Our bilingual LM corpus is counted from the training text, which overlaps ~77% of this FLEURS dev slice, so the 11.7 → 6.4 improvement is substantially the language model recognising sentences it was built from. The unfused English number (11.7) is the honest one; Twi-read 9.1 → 8.0 and conversational 26.3 → 25.8 rest on non-overlapping corpora and stand.
The defensible headline is the phone-band conversational result: BibleTTS 9.1 clean / 9.5 telephony, WAXAL 26.3 clean / 27.9 telephony — measured on named, held-out sets with no train overlap.
Phone robustness is the headline: conversational-phone WER 33.0 → 27.9 (−15% rel),
read-phone 11.7 → 9.5; clean held/improved. BibleTTS phone (9.5) ≈ clean (9.1) — nearly
codec-invariant, which an 8 kHz call pipeline needs. A bilingual KenLM 3-gram shallow fusion
(pyctcdecode) lowers WER further at decode time — English 11.7 → 6.4, Twi-read 9.1 → 8.0,
conversational 26.3 → 25.8 (see decode_lm_eval.py).
Training data
- Twi read: BibleTTS Asante (ɛ/ɔ-corrected) + Ashesi/Nokwary Financial Inclusion + Common Voice Twi.
- Conversational + code-switch: WAXAL Akan (
google/WaxalNLP,aka_asr, CC-BY-4.0). - English: FLEURS
en_us(CC-BY). - ~109 h, char-level CTC, ɛ/ɔ preserved, NFC-normalized, untoned.
Training
w2v-BERT 2.0 CTC, 12 epochs on 8×A100, batch 4, lr 1e-4, MAX_S=25. Augmentation
(TUBASTT_AUG_PROB=0.4): per-utterance 8 kHz G.711 μ-law round-trip + SpecAugment on train only.
Code: models/asr/tubastt/. Per-source numbers above are the reported results; we do not quote a
single combined dev WER, because our training-time eval metric pads accumulated logits in a way
that inflates it — the per-source evaluation is the trustworthy measurement.
Usage
import torch, soundfile as sf
from transformers import Wav2Vec2BertForCTC, Wav2Vec2BertProcessor
proc = Wav2Vec2BertProcessor.from_pretrained("neriqlabs/tubastt-twi-en-asr-w2vbert-v0.5")
model = Wav2Vec2BertForCTC.from_pretrained("neriqlabs/tubastt-twi-en-asr-w2vbert-v0.5").eval()
wav, sr = sf.read("utt.wav") # 16 kHz mono
feats = proc.feature_extractor(wav, sampling_rate=16000, return_tensors="pt").input_features
print(proc.batch_decode(model(feats).logits.argmax(-1))[0])
Limitations & next
- Conversational WER (~26% clean / 28% phone) still above read speech — the honest production bar. Next lever: pseudo-label self-training on unlabeled Ghanaian audio (UGSpeech Akan ~1000 h).
- WAXAL is Akan (Twi is a dialect); slight dialectal drift vs pure Asante Twi.
- English accent skews to FLEURS (generic); Ghanaian-English polish planned.
Base: facebook/w2v-bert-2.0 (MIT). Built by Neriqlabs.