Vietnamese ASR — FastConformer-Transducer 114M

A Vietnamese speech-to-text model built for conversational / call-center (telephony) audio. It is adapted from NVIDIA's English stt_en_fastconformer_transducer_large to Vietnamese via vocabulary adaptation and a 3-stage curriculum over ~14 public Vietnamese datasets, then hardened for robust digit-string reading on 8 kHz telephony.

• Architecture: FastConformer-Transducer (RNNT), ~114M parameters
• Language: Vietnamese (vi)
• Base model: nvidia/stt_en_fastconformer_transducer_large (CC-BY-4.0)
• Framework: NVIDIA NeMo

Highlights

Two things this model was built to demonstrate, both aimed at cold-start, noisy, narrow-domain settings where labeled data is scarce:

1. Vocabulary extension without retraining from scratch

The Vietnamese vocabulary/tokenizer was extended in place (new tokens grafted onto the existing model) rather than resetting the decoder and retraining from zero. Only a small number of vocabulary-dependent tensors need updating, so new symbols (e.g. f / j / w / z for loanwords) can be added cheaply. This makes continued, incremental training toward mixed-language content practical instead of a full re-train each time.

2. An STT ↔ TTS mutual-bootstrapping loop

Training data for hard, narrow domains was generated with a loop between STT and TTS:

• a human seeds topics and a few sample clips to clone voices from,
• a voice-cloning TTS synthesizes many in-domain utterances (e.g. connected-digit read-backs),
• the audio is degraded to 8 kHz telephony conditions,
• an STT self-check gate keeps only clips read back correctly,
• the resulting set fine-tunes the STT — which in turn makes a better gate.

This produces controlled, labeled, in-domain data without needing real production recordings, which is exactly the constraint in cold-start and privacy-restricted settings.

Results

General Vietnamese WER (9 public test sets, normalize_vi)

Digit robustness on 8 kHz telephony

The digit-hardening step targets long spoken number strings (phone, ID, card numbers) read back over 8 kHz telephony — the highest-friction failure in call-center audio.

• Digit recall on a controlled 8 kHz digit eval: 0.52 → 0.92
• Collapse (model emitting an unknown token instead of digits): eliminated
• Non-digit performance is preserved (no measurable forgetting).

Numbers on real call recordings are directional only (small labeled sample); controlled synthetic eval is used for reliable measurement.

Intended use

• Vietnamese conversational / call-center transcription, especially where callers read out numbers and identifiers over a phone line.
• Offline transcription of 8 kHz / 16 kHz Vietnamese speech.

Limitations

• Optimized for telephony/conversational Vietnamese; very formal read speech (news) is weaker.
• Domains unseen in training (e.g. medical terminology) degrade.
• Real-world call numbers were measured on a small labeled set; treat production metrics as indicative.

How to use

import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.from_pretrained("kyle/vi-asr-fastconformer-114m")
# 16 kHz mono wav; the model also handles 8 kHz telephony audio
hyp = model.transcribe(["sample_vi.wav"])
print(hyp)

Training data

All training data is public. Vietnamese sets used across the curriculum include: VIVOS, Common Voice (vi), FLEURS (vi), VLSP2020, LSVSC, FOSD (FPT Open Speech Dataset), Bud500, VietSuperSpeech, and VietMed (held-out probe). Digit-robustness data was synthesized with a voice-cloning TTS as described above.

License & attribution

• Released under CC-BY-4.0, inheriting the license of the base model.
• Derived from NVIDIA stt_en_fastconformer_transducer_large (CC-BY-4.0); please retain NVIDIA attribution.
• Built with NVIDIA NeMo.

Acknowledgements

Thanks to the open Vietnamese ASR dataset community and to NVIDIA for the FastConformer base model and the NeMo framework.