XLSR-TIMIT-B0: Fine-tuned on TIMIT for Phonemic Transcription
This model leverages the pretrained checkpoint ginic/data_seed_4_wav2vec2-large-xlsr-buckeye-ipa and is fine-tuned on the TIMIT Darpa English Corpus to transcribe audio into phonemic representations for the English language.
All code is available on Github
This model outperforms all current xlsr IPA transcription models for English
Performance
- Training Loss: 1.254
- Validation Loss: 0.267
- Test Results (TIMIT test set):
- Average Weighted Distance: 13.309375
- Standard Deviation (Weighted Distance): 9.87
- Average Character Error Rate (CER): 0.113
- Standard Deviation (CER): 0.06
Model Information
- Number of Epochs: 40
- Learning Rate: 8e-5
- Optimizer: Adam
- Datasets Used: TIMIT, Darpa English Corpus
Example Outputs
-
Prediction:
lizteɪkðɪsdɹɾiteɪbklɔθiðiklinizfɹmi
Ground Truth:lizteɪkðɪsdɹɾiteɪbəklɔtiðiklinizfɹmi
Weighted Feature Edit Distance: 7.875
CER: 0.0556 -
Prediction:
ɹænmʌðɹʔaʊtɹuhɹʔʌpɹɪŋiɾimpɛɾikoʊts
Ground Truth:ɹænmʌðɹʔaʊtɹuhɹʔʌpɹɪŋiŋinpɛɾikoʊts
Weighted Feature Edit Distance: 2.375
CER: 0.0588
Limitations
This phonemic transcription model is fine-tuned on an English speech corpus that does not encompass all dialects and languages. We acknowledge that it may significantly underperform for any unseen languages. We aim to release models and datasets that better serve all populations and languages in the future.
Usage
To transcribe audio files, this model can be used as follows:
from transformers import AutoModelForCTC, AutoProcessor
import torch
# Load model and processor
model = AutoModelForCTC.from_pretrained("KoelLabs/xlsr-timit-b0")
processor = AutoProcessor.from_pretrained("KoelLabs/xlsr-timit-b0")
# Prepare input
audio_input = "path_to_your_audio_file.wav" # Replace with your file
input_values = processor(audio_input, return_tensors="pt", sampling_rate=16000).input_values
# Retrieve logits
with torch.no_grad():
logits = model(input_values).logits
# Decode predictions
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)
print(transcription)