Nemotron 3.5 ASR Vietnamese/English Fine-Tune
This folder is a complete native Hugging Face Transformers RNNT model, including
the fine-tuned acoustic encoder, RNNT prediction network, joint network, tokenizer,
and processor. It is a conversion of a NeMo fine-tuned
nvidia/nemotron-3.5-asr-streaming-0.6b checkpoint.
The original checkpoint was trained in NeMo and exported from:
FastConformer-Transducer-BPE-Prompt-Streaming.nemo
It can be loaded with AutoProcessor and AutoModelForRNNT.
Usage
import torch
import soundfile as sf
from transformers import AutoModelForRNNT, AutoProcessor
MODEL_ID = "Nampfiev1995/nemotron35-encoder-only-vi-asr"
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
processor.set_num_lookahead_tokens(13)
model = AutoModelForRNNT.from_pretrained(
MODEL_ID,
trust_remote_code=True,
device_map="cuda:0",
).eval()
audio, sr = sf.read("audio.wav", dtype="float32")
if audio.ndim > 1:
audio = audio.mean(axis=1)
inputs = processor(
audio,
sampling_rate=sr,
return_tensors="pt",
language="vi-VN", # use "en-US" for English, or "auto" if language is unknown
)
inputs["num_lookahead_tokens"] = 13
dtype = next(model.parameters()).dtype
inputs = {
key: value.to(model.device, dtype=dtype) if torch.is_floating_point(value) else value.to(model.device)
for key, value in inputs.items()
}
with torch.inference_mode():
generated = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(generated.sequences[0], skip_special_tokens=True).strip())
Important settings:
- Use
processor.set_num_lookahead_tokens(13)and passnum_lookahead_tokens=13. - Use
max_new_tokens=512; smaller caps may truncate long utterances. - Prefer explicit language prompts such as
vi-VNoren-USwhen known.autoworks but was slightly worse.
Results Before And After Fine-Tuning
Vietnamese scores below are normalized with the repo ASR normalizer, including spoken-number
handling. The base model is nvidia/nemotron-3.5-asr-streaming-0.6b. The fine-tuned model is this
encoder fine-tune with its RNNT decoder converted to native Transformers. The repository name may
contain encoder-only for historical reasons, but the uploaded model is loadable with
AutoModelForRNNT.
| Dataset | Mode | WER | CER | Exact |
|---|---|---|---|---|
| PW733 406 | Base NeMo streaming [56,13] | 35.62% | 21.18% | 23.15% |
| PW733 406 | Fine-tuned NeMo streaming [56,13] | 13.65% | 7.20% | 56.90% |
| PW733 406 | Fine-tuned native Transformers, lookahead 13 | 14.82% | 8.34% | n/a |
| LSVSC 1000 | Base NeMo streaming [56,13] | 11.18% | 8.00% | 28.10% |
| LSVSC 1000 | Fine-tuned NeMo streaming [56,13] | 7.69% | 4.61% | 33.90% |
| LSVSC 1000 | Fine-tuned native Transformers, lookahead 13 | 7.62% | 4.55% | 34.10% |
| HeySQuAD 100 curated local manifest | Native Transformers, en-US | 11.89% | 5.25% | n/a |
| HeySQuAD 100 curated local manifest | Native Transformers, auto | 12.18% | 5.41% | n/a |
| HeySQuAD remote validation, first 100 | Native Transformers, en-US | 22.56% | 8.46% | n/a |
| HeySQuAD remote validation, first 200 | Native Transformers, en-US | 27.29% | n/a | n/a |
Summary:
- PW733 normalized WER improved from
35.62%to14.82%on the reproducible 406-file evaluation command below (342 / 2308word errors;836 / 10023character errors). - LSVSC normalized WER improved from
11.18%to about7.6%. - The converted native Transformers checkpoint matches the fine-tuned NeMo checkpoint when evaluated
with
num_lookahead_tokens=13andmax_new_tokens=512. - The
11.89%English score uses a curated local 100-sample manifest. It must not be compared directly with the remote validation split, which produced about20-27%WER in the checks above.
Upload
huggingface-cli upload \
YOUR_USERNAME/YOUR_REPO_NAME \
checkpoints/hf_nemotron35_encoder_only_0992_from_nemo \
. \
--repo-type model