MMS Ewe (BibleTTS)

Ewe speech recognition. The ewe adapter of facebook/mms-1b-all fine-tuned on BibleTTS Ewe.

Part of LIBOG, a free speech toolkit for Ghanaian languages. Its sibling is mms-asante-twi-bibletts.

⚠️ Non-commercial only

This model inherits CC BY-NC 4.0 from facebook/mms-1b-all. It may not be used commercially.

The training data, BibleTTS, is CC BY-SA 4.0, and BY-SA and BY-NC are not compatible licences. Whether trained weights constitute a derivative of their training data is unsettled law, so this is flagged rather than resolved. If you need a commercially usable Ewe recogniser, start from a permissively licensed base instead.

Results

Test split, 66 clips, scored with jiwer after Unicode NFC, lowercasing, and removing punctuation:

Both rows were measured in the same session on the same audio through the same normaliser. No figure here is carried over from another run.

Read the comparison, not the absolute number. The unmodified base already scores 13.3% on Ewe, about twice as well as it scores on Akan. Most of the distance between 6.9% here and 17.2% on the Twi sibling is that head start, not a better-executed run.

Tone is not measured. The normaliser strips the combining tone marks U+0300/0301/0303 from reference and hypothesis alike, so these figures say nothing about whether the model gets tone right — even though the ewe vocabulary, unlike Akan's, can represent it.

⚠️ This is a scripture transcriber

6.9% is measured on Bible readings by the single narrator the model trained on. Do not read it as general Ewe speech recognition, and do not assume it survives contact with spontaneous speech.

Measured off domain: 50.4% WER, on 150 WAXAL clips from 150 speakers the model has never heard, against 51.5% for unmodified mms-1b-all with the same adapter. That 1.1-point gap has a 95% bootstrap confidence interval of [−0.37, +2.55], which crosses zero, and character error is 0.2 points worse than the base.

So on general Ewe speech this model is not measurably better than the one it started from. It is not worse either, and it does not collapse — zero empty hypotheses across both arms. But if your audio is not read scripture, the honest position is that stock facebook/mms-1b-all with the ewe adapter will serve you about as well.

The Twi sibling differs here: its off-domain gap is 3.4 points in its own favour. The likely reason is that Meta's ewe adapter simply starts stronger than its aka one, leaving less room to improve on — but that is a hypothesis, not a finding.

For general Ghanaian-language speech recognition, Khaya AI from GhanaNLP is trained on multi-speaker, multi-domain data.

Usage

import torch, soundfile as sf
from transformers import AutoProcessor, Wav2Vec2ForCTC

repo = "danieldzikunuofmarvel/mms-ewe-bibletts"
processor = AutoProcessor.from_pretrained(repo)
model = Wav2Vec2ForCTC.from_pretrained(repo).eval()

audio, sr = sf.read("clip.wav")          # 16 kHz mono
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    ids = torch.argmax(model(**inputs).logits, dim=-1)[0]
print(processor.decode(ids))

The adapter is already merged into these weights, so do not pass target_lang. Output is lowercase and carries no sentence punctuation: the MMS Ewe vocabulary is 55 character-level tokens, and the only non-letters in it are ', -, the word delimiter, and the three combining tone marks. An apostrophe or hyphen can appear in output; a full stop or comma never will.

Training

Best checkpoint selected on dev WER rather than loss.

The detail that matters if you copy this recipe

MMS's character vocabularies contain no sentence punctuation. Training on transcripts that carry it makes every comma and full stop an <unk>, and the model learns to predict a token holding no information. On the Twi sibling that cost a whole GPU run: starting loss 3.03 instead of 0.65, test WER 28.9% instead of 17.2%.

Labels here are filtered to the tokenizer's vocabulary before training, which removes about 4% of characters. The check is cheap — count <unk> in a sample of encoded labels and assert it is zero before spending on GPU time.

The Ewe vocabulary's 55 tokens cover Ewe orthography in full, including ɖ ƒ ŋ ɣ ʋ, the nasal vowels, and the three combining tone marks — so tone survives into the training labels, which it cannot do in the 35-token Akan vocabulary. The Ewe BibleTTS transcripts also needed no repair, where the Asante Twi release had ɛ and ɔ stripped from the text across the whole corpus and had to be repaired and republished before anything could train on it.

Attribution

• Base model: MMS, Pratap et al., Meta AI. CC BY-NC 4.0.
• Speech data: BibleTTS, Meyer et al., Interspeech 2022. CC BY-SA 4.0.