ghana-speech-phoneme-asr

A single CTC model that transcribes speech in 42 Ghanaian and West African languages directly into IPA phonemes, with punctuation.

Built for phoneme-level forced alignment and pronunciation work: the output is a sequence of IPA units rather than orthographic words, so it can be aligned to audio, compared across languages, or used as a TTS front-end target.

Quick start (sherpa-onnx)

import sherpa_onnx, soundfile as sf

rec = sherpa_onnx.OfflineRecognizer.from_omnilingual_asr_ctc(
    model="onnx/model.int8.onnx",     # or model.onnx for fp32
    tokens="onnx/tokens.txt",
    num_threads=4,
)

wav, sr = sf.read("utterance.wav", dtype="float32")   # 16 kHz mono
s = rec.create_stream()
s.accept_waveform(sr, wav)
rec.decode_stream(s)

print(s.result.tokens)   # ['m', 'f', 'ia', 's', 'e', 'n', 'o', 'o', 'ɲ', ...]
print(s.result.text)     # run together

result.tokens is the list you want — each element is one phoneme unit. Several units are multi-character (kʰ, k͡p, t͡ʃ, nʷ), so never split the text on characters; that would turn one sound into two.

Files

onnx/model.onnx           fp32 ONNX, 1.3 GB
onnx/model.int8.onnx      int8 quantised, 349 MB  <- use this for deployment
onnx/tokens.txt           176 tokens, real IPA symbols
checkpoint/               fairseq2 checkpoint (model + optimizer + trainer state)
tokenizer/phoneme_spm.model    SentencePiece char model over the phoneme proxies
tokenizer/phonemes.json        proxy codepoint <-> IPA mapping, and unit frequencies
tokenizer/inventories.json     each language's own phoneme inventory
config/ctc-ghana-ipa.yaml      the training recipe config
config/model_arch.yaml         model architecture
config/ghana_ipa.yaml          fairseq2 asset cards (tokenizer + dataset)
config/language_distribution.tsv   hours per language, drives temperature sampling
eval_results.json         per-language evaluation

Results

Evaluated on held-out dev audio (100 clips per language) decoded through sherpa-onnx. UER is unit error rate over phonemes; the phonemes-only column strips punctuation from both sides.

Best and worst languages, phonemes-only:

Full per-language numbers are in eval_results.json.

Read these per-language, not as one number. The spread is large and does not track data volume — Dagaare has 15 h and scores 13.0%, Hausa has 152 h and scores 38.0%. Phonemisation consistency appears to matter more than corpus size.

Ninkare, Hausa and Kabiye are outliers worth treating with suspicion. All three had heavily repaired rule data upstream, and their targets may still be inconsistent, which would cap achievable accuracy regardless of the model. Verify before relying on them.

Punctuation

Punctuation is included in the output (27 of the 172 units). It costs 1.25 UER points overall, and the model gets it right about 62% of the time — it is clearly using pauses and intonation. Quotation marks are the least predictable, since nothing in the audio marks where a quote opens.

How the phoneme vocabulary works

Phoneme units are frequently multi-character (kʰ, k͡p, t͡ʃ). The tokenizer family used by omniASR segments by character, which would split those apart, so each unit was mapped to a single Private Use Area codepoint (U+E000…) for training. onnx/tokens.txt maps them back, so the deployed model emits real IPA and nothing downstream ever sees the proxies.

tokenizer/phonemes.json holds the mapping if you need to regenerate or extend the vocabulary.

Per-language output restriction

The model shares one 172-unit inventory across all 42 languages, but any single language uses only 25–52 of them (24% of the total on average). If you know the language in advance, you can restrict the output to that language's inventory — tokenizer/inventories.json lists them — by slicing the final projection before export. This makes it impossible for the model to emit another language's phonemes.

We did not do this by default, because the shared inventory is what gives the low-resource languages their transfer benefit, and the model rarely strays out of inventory in practice.

Training

Trained with omnilingual-asr's wav2vec2.asr recipe on a single H200.

• CTC head trained from scratch on omniASR_W2V_300M (encoder not frozen)
• lr 1e-5, bf16 autocast, ~480 s of audio per step
• Temperature-sampled across languages (beta_language=0.5) so 15-hour languages are not swamped by 200-hour ones
• Best checkpoint at step 20,000, selected on validation UER

UER progression: 100% → 45.5% (step 4k) → 27.6% (9k) → 22.1% (13.5k) → 18.88% (20k).

Note the 18.88% validation figure and the 16.64% evaluation figure differ because they use different dev sampling; the per-language table is the one to trust.

Resuming or fine-tuning

checkpoint/ contains model, optimizer and trainer state, so training can resume exactly. The prepared training corpus is published separately as ghananlpcommunity/ghana-speech-ipa-asr-ready — already in the partitioned parquet layout the recipe expects.

Code and worked examples: GhanaNLP/ghana-phoneme-asr

Known limitations

• Digits are dropped, not verbalised. Numbers in the audio produce no phonemes.
• Language is not identified. The model transcribes phonetically without knowing which language it heard; pair it with a language ID model if you need that.
• Punctuation is approximate, especially quotation marks.
• Clips longer than 40 s are outside the range the base model accepts.
• 10 of the 42 languages use phoneme rules borrowed from a related language, because none existed for them. They perform well, but the phoneme values are plausible rather than authoritative. See the tier field in the phoneme dataset.

Sources

• Audio: ghananlpcommunity/ghana-speech
• Phoneme targets: ghananlpcommunity/ghana-speech-phonemes
• Phonemisation: GhanaNLP/ghana-g2p
• Base model: facebookresearch/omnilingual-asr (Apache-2.0)

Licence

CC BY-NC 4.0, following the source audio corpus.