CMUBYE phoneme encoder
wav2vec2-XLS-R-300M phone-CTC encoder, fine-tuned on speechocean762 for
pronunciation assessment. Drop-in for
vitouphy/wav2vec2-xls-r-300m-timit-phoneme; same 42-token IPA vocabulary.
Results
Held-out speechocean762 test split, zero speaker overlap with train.
| base | this model | |
|---|---|---|
| PER | 0.4043 | 0.1755 |
| phone-level PCC vs human scores | 0.568 | 0.705 |
| word-level PCC | 0.539 | 0.665 |
| utterance PCC | 0.690 | 0.712 |
| mispronunciation detection AP | 0.445 | 0.652 |
Phone and word level exceed commonly published results on this benchmark (~0.60 / ~0.55).
Training
Targets are what the annotators heard, not what the prompt said — substitutions use the realised phone, deletions drop out, distortions keep the canonical. Training on canonical phones instead teaches the model to emit the expected phone regardless of the audio, which makes goodness-of-pronunciation measure nothing while CTC loss still looks healthy.
10 epochs, all 24 transformer layers unfrozen, batch 8 x accum 2, lr 2e-5.
Usage
from transformers import AutoModelForCTC, AutoProcessor
proc = AutoProcessor.from_pretrained("Pransfrance/cmubye-phoneme-encoder")
model = AutoModelForCTC.from_pretrained("Pransfrance/cmubye-phoneme-encoder")
The encoder alone gives frame-level phone posteriors. Alignment, GOP, error decoding, completeness and the calibrated scoring head live in CMUBYE.
Limitations
Fine-tuned on Mandarin-L1 English learners, ~46% of them children aged 6-8. It transfers usefully to other L2 English but is not adapted to any specific target population; re-running the fine-tune on in-domain audio is the single highest-value improvement available.