WavLM-Large French Phoneme Recognizer
microsoft/wavlm-large fine-tuned with a CTC head for French phoneme recognition,
outputting IPA symbols directly from audio without an intermediate word-level step.
PER 3.50% (dev) / 4.78% (test) on the full Common Voice 19.0 French splits, under the common evaluation protocol described below.
This model was built as a transcription front-end for phoneme alignment. It is one of the systems benchmarked in "Comparing Phoneme Alignment Pipelines Across Spontaneous and Pathological French Speech" (SLT 2026), where direct phoneme recognition was found to consistently improve alignment over a conventional ASR + grapheme-to-phoneme pipeline.
Usage
import torch, librosa, soundfile as sf
from transformers import AutoProcessor, WavLMForCTC
model_id = "imenba6/wavlm-large-fr-phoneme-recognition"
processor = AutoProcessor.from_pretrained(model_id)
model = WavLMForCTC.from_pretrained(model_id).eval()
wav, sr = sf.read("example.wav", dtype="float32")
if wav.ndim > 1:
wav = wav.mean(1) # to mono
if sr != 16000:
wav = librosa.resample(wav, orig_sr=sr, target_sr=16000); sr = 16000
inputs = processor(wav, sampling_rate=sr, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits # (1, frames, 39)
print(processor.batch_decode(logits.argmax(-1))[0])
# -> "ɔ̃ d i k ə i l i ɡ a l a ʁ e ʒ j ɔ̃ d ə b o s"
Audio must be 16 kHz mono; the feature extractor applies zero-mean/unit-variance
normalization (do_normalize: true).
Loading with torchaudio.load also works, but torchaudio >= 2.9 requires the extra
torchcodec package for it, so soundfile is used above to keep dependencies minimal.
The encoder outputs one frame per 20 ms (49.8 Hz), which sets the granularity of any boundary you derive from these posteriors.
Forced alignment
For phoneme boundaries rather than just the sequence, use the frame-level CTC posteriors
with a Viterbi forced aligner (e.g. torchaudio.functional.forced_align), or feed the
predicted phoneme string to MFA. In the accompanying paper, MFA produced more accurate
boundaries than CTC alignment across all conditions, so the recommended pipeline is
this model → MFA. See alignment_models/align_viterbi.py and mfa/ in the repo.
Phoneme inventory
37 tokens: 35 French IPA phonemes plus [UNK] and [PAD] (the CTC blank),
with <s>/</s> as added tokens (vocab size 39).
a b d e f i j k l m n o p s t u v w y z ø ŋ œ œ̃ ɑ̃ ɔ ɔ̃ ə ɛ ɛ̃ ɡ ɲ ʁ ʃ ʒ
Phonemes are space-separated in the output. Note there is no word delimiter token
(word_delimiter_token: null): the model emits a flat phoneme stream with no word
boundaries.
Training
| Base model | microsoft/wavlm-large (24 layers, hidden size 1024) |
| Data | Common Voice French 19.0 (cv-corpus-19.0-2024-09-13) |
| Train / dev | 507,972 / 14,325 utterances |
| Utterance length | 1–20 s |
| Epochs | 15 (476,220 steps) |
| Batch size | 8 × 2 grad. accumulation = 16 effective |
| Learning rate | 1e-5, 1,500 warmup steps, linear decay |
| Loss | CTC (mean reduction, zero_infinity=True) |
| SpecAugment | mask_time_prob=0.075 |
| Precision | see training_args.bin in the training repo |
Target phoneme sequences were produced by grapheme-to-phoneme conversion of the Common Voice transcripts, so the training targets are themselves automatic.
Evaluation: full Common Voice dev and test
Both this model and the companion Whisper phoneme recognizer were re-evaluated on the full Common Voice 19.0 French dev and test splits under one shared protocol, so the two are directly comparable:
| Model | dev PER | test PER |
|---|---|---|
| WavLM-large (this model) | 3.504% | 4.776% |
| Whisper-large-v3 encoder | 4.600% | 6.525% |
| Split | Utterances | Reference phones | Edits (this model) |
|---|---|---|---|
| dev | 14,574 | 551,967 | 19,341 |
| test | 14,771 | 551,241 | 26,327 |
Both models were scored on the identical utterance set and identical references within each split.
Protocol
- References produced with the same G2P used for training:
phonemizer+ espeak,fr-fr,strip=True, no stress, length markːremoved. Verified byte-identical to the original training references on all 14,325 overlapping dev utterances. - espeak word delimiters
|and language-switch markers(fr)/(en)removed. œ̃merged intoɛ̃in both reference and hypothesis. This is required for a fair comparison: the Whisper model has noœ̃output unit at all, because its training references had the merge already applied.- An utterance is kept only if every remaining reference symbol falls in the 34-phoneme inventory shared by both models. This drops utterances containing English phones — the same filtering the original training data preparation applied. 90.1% of dev and 91.4% of test survive.
- PER = Levenshtein distance over phoneme sequences / reference length, aggregated over the split.
Evaluation caveats
3.504% / 4.776% figures are the full splits under the common protocol above.
- Reference phonemes are G2P-derived, not manually verified, so PER measures agreement with an automatic reference rather than with human phonetic transcription.
- Common Voice is read speech. Expect substantially higher error rates on spontaneous and pathological speech — this is exactly what the accompanying paper measures. On those corpora, performance degrades considerably relative to read speech, and alignment quality degrades with it.
License
MIT, inheriting from microsoft/wavlm-large. Training data is Common Voice 19.0 (CC0).
Citation
Paper under review at SLT 2026; citation details will be added on acceptance.
@inproceedings{phoneme_alignment_slt2026,
title = {Comparing Phoneme Alignment Pipelines Across Spontaneous and
Pathological French Speech},
author = {Imen Ben-Amor, Jean-Luc Rouas, Charles Brazier},
booktitle = {IEEE Spoken Language Technology Workshop (SLT)},
year = {2026}
}