nemotron-3.5-asr-streaming-0.6b-ht-v2

Haitian Creole (Kreyòl Ayisyen) fine-tune of nvidia/nemotron-3.5-asr-streaming-0.6b — a 0.6B cache-aware streaming FastConformer RNN-T with language-ID prompt conditioning.

What changed

The base model conditions on language through a 128-slot one-hot prompt vector, but its prompt_dictionary only assigns indices 0–104. This fine-tune claims free slot 105 for ht-HT, warm-started from fr-FR.

Pass target_lang="ht-HT" to get Haitian Creole. The slot was initialized from fr-FR because Haitian Creole is French-lexified Latin script, giving the new language a useful prior instead of starting from noise.

The pretrained BPE tokenizer (vocab 13088) is reused unchanged, so the RNN-T joint keeps its pretrained weights.

⚠️ This is a Creole-only model — the other languages are NOT preserved

Claiming a free slot protects the other language columns from gradient, and they are in fact untouched by it. It does not protect model behaviour: the encoder, decoder and joint are shared by every language and were fine-tuned on Haitian Creole only, so the backbone drifted. Measured on this checkpoint, on Haitian Creole audio:

The control matters: the base model steers correctly on the same audio, so this is a real change in this checkpoint rather than an artifact of the audio being Creole.

Root cause: slot 105 was warm-started from fr-FR and never diverged from it (cos = 0.99993 after training), so both prompts feed nearly the same vector into a backbone that now only knows Creole.

Do not use this model for any language other than Haitian Creole. If you need the base model's multilingual coverage, use nvidia/nemotron-3.5-asr-streaming-0.6b directly. Preserving it through a fine-tune requires mixing a replay set of the original languages into training.

Training data

• Total: 60.579 h of transcribed Haitian Creole speech, 16 kHz mono.

• jsbeaudry__m1-collect — 5.267 h

• jsbeaudry__bible-chapters-data — 3.789 h

• jsbeaudry__media-social-2 — 3.659 h

• jsbeaudry__learn-the-numbers — 1.405 h

• jsbeaudry__justice-in-creole — 1.998 h

• jsbeaudry__finance-in-creole — 1.229 h

• jsbeaudry__history-data — 1.219 h

• jsbeaudry__ecommerce-creole — 0.49 h

• jsbeaudry__new-data-format — 0.025 h

• jsbeaudry__priye-kreyol — 0.03 h

• jsbeaudry__cmu_haitian_creole_speech — 15.884 h

• jsbeaudry__openai-tts-data — 7.125 h

• jsbeaudry__ggl-tts-data — 0.675 h

• radio-haiti-inter — 17.782 h

Pseudo-labeled data

17.782 h of the training data is real audio carrying machine-generated transcripts rather than human ones. Those clips were filtered on two independent signals — the source corpus's own per-segment confidence, and per-segment CER agreement with an earlier checkpoint of this model — and were excluded from validation and test, so the benchmark above is measured entirely against human-labelled recordings. Residual label noise in the training set is expected.

Benchmark

Benchmark

• Test clips: 332
• Scoring: exact

Baseline = untouched base model prompted with its closest existing language; fine-tuned = the new ht-HT slot.

Usage

import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.restore_from("nemotron-3.5-asr-streaming-0.6b-ht-v2.nemo")
model.encoder.set_default_att_context_size([56, 3])   # 320 ms chunks
print(model.transcribe(["clip.wav"], target_lang="ht-HT"))

Streaming inference (matches how the benchmark above was measured):

python $NEMO_DIR/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
  model_path=nemotron-3.5-asr-streaming-0.6b-ht-v2.nemo \
  dataset_manifest=test.json \
  target_lang=ht-HT \
  att_context_size="[56,3]" \
  decoder_type=rnnt pad_and_drop_preencoded=true strip_lang_tags=false

att_context_size left context must be 56 — it matches the encoder's sliding_window: 57. Right context selects latency: 0→80 ms, 3→320 ms, 6→560 ms, 13→1120 ms.

Loading with transformers

Not yet supported: this repo ships the NeMo checkpoint, and converting it to safetensors requires NVIDIA's .nemo→HF converter. Use NeMo as shown above.

Known limitations

• Creole only — see the warning above.
• Word segmentation dominates the remaining errors. CER is roughly a third of WER (≈4.8% vs ≈15.7% at 320 ms), which is the signature of boundary errors rather than acoustic ones: Si l→Sil, ou mèt→Omèt, bò dlo→bòd lo. Proper nouns and numbers are the other common failures (Dariyis→Dariyich, 120→12). A decode-time phrase-boosting list or an n-gram LM would target this directly.
• Training data is partly synthetic. TTS-generated speech was used for training only and excluded from the validation and test splits, so the numbers above reflect real recordings.
• Some sources are machine-transcribed. Clips whose transcripts exceeded 25 characters/second — degenerate repetition loops such as pou pou pou pou … — were filtered out, but milder label noise below that threshold may remain. Where a source is machine-transcribed by design (see "Pseudo-labeled data" above), it additionally had to clear a confidence threshold and agree with an earlier checkpoint of this model before it was allowed to train.
• Clips longer than 18 s were excluded from training due to a GPU memory limit on a 46 GB card, not for data-quality reasons. Long-utterance behaviour is therefore less well covered.
• Training stopped early, at roughly epoch 5 of a planned 15, while validation WER was still improving. This checkpoint is not the ceiling for this data.

License

Inherits OpenMDW-1.1 from the base model. Third-party training corpora keep their own licences — see Attribution below.

Attribution

Radio Haiti-Inter

Havard, W.; Ziane, R.; Menclé, M.; Coavoux, M.; Lecouteux, B.; Schang, E. Radio Haïti-Inter (sample-1). Zenodo, 2025. Laboratoire Ligérien de Linguistique (Université d'Orléans) and Laboratoire d'Informatique de Grenoble (Université Grenoble Alpes), ANR CREAM (ANR-20-CE38-0006). doi:10.5281/zenodo.17818122 — CC-BY-4.0. Used here with modifications: re-segmented to the 1–15 s window, filtered by confidence and by agreement with an earlier checkpoint of this model, and Creole clitics restyled to the apostrophe spelling used by the rest of the corpus.

The transcripts in that release are machine-generated. The model that produced them is described in:

Havard, W. N.; Govain, R.; Lecouteux, B.; Schang, E. Self-Supervised Models of Speech Processing for Haitian Creole. Interspeech 2025, 4018–4022. doi:10.21437/Interspeech.2025-1852