whisper-large-v3-sk
Slovak fine-tune of openai/whisper-large-v3 by the KInIT team. Full parameter fine-tuning on a curated Slovak speech corpus, with noise augmentation for robustness to real-world recording conditions. Listed in SLAIH, a catalog of Slovak NLP resources.
Model Details
| Property | Value |
|---|---|
| Base model | openai/whisper-large-v3 |
| Parameters | ~1.55B |
| Architecture | Whisper encoder-decoder |
| Fine-tuning method | Full fine-tuning |
| Language | Slovak (sk) |
| Task | Automatic Speech Recognition |
| License | Apache 2.0 |
Intended Use
This model is intended for Slovak automatic speech recognition across a range of domains and recording conditions.
Out-of-scope: Non-Slovak audio, real-time streaming without appropriate chunking, safety-critical transcription without human review.
Evaluation
Word Error Rate (WER) and Character Error Rate (CER), lower is better. Measured on two Slovak eval sets:
- CV24 - Common Voice 24.0 test split (5,239 samples, public)
- Internal - held-out KInIT set (9,317 samples, stratified by domain and speaker gender, one third clean and two thirds noise-augmented; not public)
Before vs. after fine-tuning
| Model | CV24 WER ↓ | CV24 CER ↓ | Internal WER ↓ | Internal CER ↓ |
|---|---|---|---|---|
| kinit/whisper-large-v3-sk | 7.68% | 2.27% | 5.67% | 3.01% |
| openai/whisper-large-v3 | 19.70% | 5.55% | 16.22% | 7.89% |
Slovak fine-tuning reduces WER by 61% on Common Voice and 65% on the internal eval set, at the cost of multilingual performance (see Limitations).
Choosing a model size
Comparison of all six Whisper sizes before and after Slovak fine-tuning. WER is measured on our internal benchmark dataset (see above):
| Model | Fine-tuned: WER ↓ | Base: WER ↓ |
|---|---|---|
| Whisper Large v3 | 5.67% | 16.22% |
| Whisper Large v3 Turbo | 5.60% | 18.35% |
| Whisper Medium | 6.28% | 29.19% |
| Whisper Small | 8.64% | 47.90% |
| Whisper Base | 14.01% | 76.38% |
| Whisper Tiny | 20.04% | 99.47% |
The full KInIT ASR collection includes all six Whisper sizes plus the NVIDIA NeMo fine-tunes (Canary, Parakeet).
Version History
- v2 (current): Added noise augmentation during fine-tuning (75% of training batches) for improved robustness to real-world recording conditions.
- v1: Original release, full fine-tuning without noise augmentation.
| Release | CV24 WER ↓ | CV24 CER ↓ | Internal WER ↓ | Internal CER ↓ |
|---|---|---|---|---|
| v2 (current) | 7.68% | 2.27% | 5.67% | 3.01% |
| v1 | 8.41% | 2.60% | 6.19% | 3.21% |
Loading a specific release
Every release is tagged: v2.0 for the current release, v1.0 for the original one. main always points at the current release, so the Usage example below loads v2.
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
repo_id = "kinit/whisper-large-v3-sk"
revision = "v2.0" # "v1.0" for the original release
model = AutoModelForSpeechSeq2Seq.from_pretrained(repo_id, revision=revision)
processor = AutoProcessor.from_pretrained(repo_id, revision=revision)
Pin revision in production code if you need reproducible results across future releases.
Training Data
Fine-tuned on an internal curated Slovak speech corpus compiled at KInIT. The corpus combines public datasets with internal KInIT recordings. Recordings containing personal data were anonymised prior to use. Samples were quality-filtered using a CER-based threshold validated against multiple ASR models. During training, 75% of batches were additionally augmented on the fly with synthetic background noise for robustness to real-world recording conditions (see Training Procedure below).
| Data sources |
|---|
| SloPalSpeech |
| Municipal council session recordings |
| Read literature |
| Mozilla Common Voice |
| TEDxSK and JumpSK Lecture Speech Corpus |
| FLEURS read speech |
| Internal KInIT recordings |
Training Procedure
| Hyperparameter | Value |
|---|---|
| Epochs | 2 |
| Learning rate | 5e-5 |
| LR scheduler | Linear with warmup |
| Optimizer | AdamW |
| Effective batch size | 64 |
| Precision | fp16 |
| Noise augmentation | Applied on the fly to 75% of training batches: phone noise, background speech, background noise, white noise, and packet loss |
| Framework | HuggingFace Transformers Seq2SeqTrainer |
Training was performed on the Devana HPC cluster.
Usage
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
model_id = "kinit/whisper-large-v3-sk"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id,
dtype=dtype,
use_safetensors=True,
).to(device)
processor = AutoProcessor.from_pretrained(model_id)
pipe = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
dtype=dtype,
device=device,
)
result = pipe("audio.wav", generate_kwargs={"language": "slovak"})
print(result["text"])
License
Limitations
- Catastrophic forgetting: Fine-tuning exclusively on Slovak data significantly degrades performance on other languages. Use the base openai/whisper-large-v3 if multilingual transcription is required.
- Performance may degrade on strongly accented, dialectal, or domain-specific speech not represented in the training data.
- Maximum reliable single-segment length is 30 seconds without chunking.
Acknowledgements
Public datasets used in training: SloPalSpeech, Mozilla Common Voice, FLEURS, and the TEDxSK and JumpSK Lecture Speech Corpus (KEMT NLP).
(Part of the) Research results was obtained using the computational resources procured in the national project National competence centre for high performance computing (project code: 311070AKF2) funded by European Regional Development Fund, EU Structural Funds Informatization of society, Operational Program Integrated Infrastructure.