Disclaimer / Notice: Details for these are in Peer Review and publications of the paper will be made available soon for more details.

xlsr-300m-mizonal3-E1-lus-v2026.06

This model is a fine-tuned version of facebook/wav2vec2-xls-r-300m on the MiZonal v3.0 dataset. Note: ~1 hour of conversational speech was added to this dataset version.

It achieves the following results on the evaluation set:

• Wer: 19.1028
• Cer: 3.7453
• Real Time Factor: 0.0024

Quick Inference

import torch
import librosa
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Wav2Vec2Processor.from_pretrained("andrewbawitlung/xlsr-300m-mizonal3-E1-lus-v2026.06")
model = Wav2Vec2ForCTC.from_pretrained("andrewbawitlung/xlsr-300m-mizonal3-E1-lus-v2026.06").to(device)

audio, sr = librosa.load("your_audio.wav", sr=16000)
input_values = processor(audio, sampling_rate=16000, return_tensors="pt").input_values.to(device)

with torch.no_grad():
    logits = model(input_values).logits

predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)[0]
print(transcription)

Model description

Experiment Configurations

This repository is part of a series of experiments. The different configurations are:

• E1 (Baseline): Standard training configuration.
• E2 (Noise): Training with background noise augmentation.
• E3 (Speed): Training with speed perturbation augmentation.
• E4 (SpecAug): Training with SpecAugment (time and frequency masking).
• E5 (Combined): Training with a combination of all augmentations.

All Models in this Family

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: OptimizerNames.ADAMW_TORCH_FUSED
• lr_scheduler_type: SchedulerType.LINEAR
• num_epochs: 8

Training results