whisper-small-vi-finetuned
A Vietnamese automatic speech recognition model, fine-tuned from
openai/whisper-small (244M parameters) on
natmin322/18k_vietnamese_voice_augmented_of_VigBigData.
Trained entirely on Kaggle across two resumed sessions within a 12-hour/day dual-T4 GPU budget.
Results
Measured against the unmodified base model on a held-out 500-sample test subset:
| Model | WER (%) | CER (%) |
|---|---|---|
openai/whisper-small | 67.33 | 45.20 |
qa2code/whisper-small-vi-finetuned | 31.74 | 19.43 |
A 53% relative reduction in Word Error Rate and 57% in Character Error Rate.
Beyond the metrics, two qualitative changes matter. The base model occasionally collapses into a repetition loop, emitting the same phrase until it hits the generation length limit; the fine-tuned model did not exhibit this on any inspected sample. It also adopted the corpus's orthographic conventions — see Output format below.
Usage
from transformers import pipeline
pipe = pipeline(
"automatic-speech-recognition",
model="qa2code/whisper-small-vi-finetuned",
device=0, # omit for CPU
)
result = pipe("audio.wav", generate_kwargs={"language": "vi", "task": "transcribe"})
print(result["text"])
Audio is resampled to 16 kHz internally. Passing language="vi" explicitly is recommended —
without it, Whisper's language detection can misfire on short or noisy clips and decode into
the wrong language.
For clips longer than 30 seconds, enable chunked long-form decoding:
result = pipe(
"long_audio.wav",
chunk_length_s=30,
stride_length_s=5,
generate_kwargs={"language": "vi", "task": "transcribe"},
)
Loading the model and processor directly:
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
processor = AutoProcessor.from_pretrained("qa2code/whisper-small-vi-finetuned")
model = AutoModelForSpeechSeq2Seq.from_pretrained(
"qa2code/whisper-small-vi-finetuned", device_map="auto"
)
Output format
The model inherits the transcription conventions of its training corpus. Output is:
- lowercase throughout — no sentence-initial capitals, no capitalised proper nouns
- without terminal punctuation — no full stops, commas, or question marks
- numbers spelled as words —
mười tám tháng mười, not18 tháng 10
This is a deliberate property of the data, not a defect, but it means the raw output is not display-ready. Downstream applications needing conventional orthography should add a punctuation-and-capitalisation restoration step.
Training data
The source dataset contains 18,005 audio–transcript pairs at 22.05 kHz. Exploratory analysis
showed a heavily right-skewed duration distribution (median 5.76 s, mean 6.92 s, tail to 80 s)
and residual <unk> markers left over from the upstream pipeline.
A cleaning pass applied three filters:
- Keep clips with duration in [1 s, 25 s] — shorter clips carry little linguistic content, longer ones exceed Whisper's 30-second receptive field and waste VRAM on padding.
- Strip
<unk>markers and collapse the resulting whitespace. - Drop transcripts shorter than two words.
Result: 18,005 → 17,169 samples (9,695 train / 4,647 validation / 2,827 test).
Training procedure
| Setting | Value |
|---|---|
| Hardware | 2 × NVIDIA T4 (Kaggle) |
| Precision | Native AMP (FP16) |
| Learning rate | 1e-5, linear schedule, 100 warmup steps |
| Optimizer | AdamW (torch fused), β=(0.9, 0.999), ε=1e-8 |
| Batch size | 8 per device × 4 gradient accumulation steps (effective 32) |
| Training steps | 3,000 (≈9.9 epochs over the training split) |
| Eval / checkpoint | every 500 steps, best-by-WER checkpoint retained |
| Seed | 42 |
Training was split across two Kaggle sessions to fit the 12-hour daily dual-T4 quota. The first
session ran to step 1,000 and pushed its checkpoint to the Hub; the second resumed from
checkpoint-1000 with max_steps raised to 3,000, restoring optimizer state, scheduler
position, and RNG state. The run is therefore continuous rather than two independent trainings.
Validation during training used a fixed 500-sample subset to bound evaluation cost.
Training results
| Training Loss | Epoch | Step | Validation Loss | WER | CER |
|---|---|---|---|---|---|
| 4.2719 | 1.6502 | 500 | 0.8749 | 37.05 | 23.19 |
| 2.7014 | 3.3003 | 1000 | 0.8595 | 35.32 | 22.21 |
Limitations
In-domain evaluation only. The 31.74 WER figure is measured on a held-out split of the same corpus the model was trained on — read speech, studio-adjacent conditions, augmented. It is not directly comparable to numbers reported on Common Voice, VIVOS, or FLEURS, and real-world performance on spontaneous speech, regional accents, telephone-bandwidth audio, or noisy environments will be meaningfully worse. Benchmark on your own data before deploying.
Undertrained by design. 3,000 steps is a modest budget for a 244M-parameter encoder–decoder, and the loss curve had not plateaued when the run was stopped by the hardware quota. This result is closer to a floor than a ceiling.
No punctuation or casing, as described in Output format.
Inherits Whisper's failure modes, including hallucination on silence and non-speech audio. Voice activity detection upstream is advisable for production use.
Directions for improvement
- Extend training past 3,000 steps — the largest single gain still available
- Add SpecAugment for regularisation
- Evaluate on out-of-domain Vietnamese benchmarks to measure generalisation
- Explore LoRA/PEFT to fit
whisper-mediumorwhisper-large-v3into the same VRAM envelope
Framework versions
- Transformers 5.0.0
- PyTorch 2.10.0+cu128
- Datasets 5.0.0
- Tokenizers 0.22.2
License
Apache 2.0, inherited from openai/whisper-small. Note that the training dataset carries its
own licence terms — check the
dataset card
before commercial use.
Citation
@misc{whisper-small-vi-finetuned,
author = {Nguyen, Quoc Anh},
title = {whisper-small-vi-finetuned: Vietnamese ASR fine-tuned from Whisper-small},
year = {2026},
url = {https://huggingface.co/qa2code/whisper-small-vi-finetuned}
}