w2v-bert-2.0-hungarian-CV24.0

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the Common Voice Scripted Speech 24.0 - Hungarian dataset. It achieves the following results on the evaluation set:

• Loss: 0.1203
• Wer: 0.0781

Evaluation data

Results on the Common Voice Scripted Speech 24.0 - Hungarian test dataset

================================================================================ EVALUATION RESULTS SUMMARY

                      Metric  Value    Std
       Word Error Rate (WER)  5.65% 11.90%
  Character Error Rate (CER)  1.47%  4.23%
   Sentence Error Rate (SER) 32.26% 46.75%
      Match Error Rate (MER)  5.42% 10.49%
 Word Information Lost (WIL)  8.50% 15.24%     

Word Information Preserved (WIP) 91.50% 15.24%

            Total Samples  12066    

================================================================================ SAMPLE ERRORS (Highest WER)

--- Sample 1 (WER: 450.00%, CER: 254.55%) --- Reference: chet greene Hypothesis: chet|gren|egy|ilyen|tisztázás|nek|áa|| Insertions: 28 Deletions: 0 Substitutions: 0

--- Sample 2 (WER: 150.00%, CER: 14.29%) --- Reference: radkersburg ut Hypothesis: ratkersburg|út Insertions: 1 Deletions: 0 Substitutions: 1

--- Sample 3 (WER: 150.00%, CER: 33.33%) --- Reference: gyolcsot sző Hypothesis: gyolcsat|szün Insertions: 2 Deletions: 0 Substitutions: 2

--- Sample 4 (WER: 133.33%, CER: 14.81%) --- Reference: universala esperanto asocio Hypothesis: universala|eszperanto|a|szócio Insertions: 4 Deletions: 0 Substitutions: 0

--- Sample 5 (WER: 100.00%, CER: 25.00%) --- Reference: kotródj Hypothesis: kotrudj Insertions: 0 Deletions: 1 Substitutions: 1

Use

Due to tokenizer metadata resolution inconsistencies in the HuggingFace hub loader, users are recommended to download the repository locally and perform inference using local_files_only=True.

git lfs install
git clone https://huggingface.co/GaborMadarasz/w2v-bert-2.0-hungarian-CV24.0

from transformers import AutoProcessor, AutoModelForCTC

model_path = "./w2v-bert-2.0-hungarian-CV24.0"

processor = AutoProcessor.from_pretrained(
    model_path,
    local_files_only=True
)

model = AutoModelForCTC.from_pretrained(
    model_path,
    local_files_only=True
)

model.eval()

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10

Training results

Framework versions

• Transformers 4.57.3
• Pytorch 2.9.0+cu128
• Datasets 4.5.0
• Tokenizers 0.22.1

Contact

gabor.madarasz@gmail.com