Parakeet RNN-T 0.6B — checkpoint soup of one run's last five epochs

An equal-weight average of the epoch 6, 7, 8, 9 and 10 checkpoints of dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout. No training was done here, and inference costs exactly what one model costs.

It is worth 0.27 CER points over the best single epoch. Constrained track: the ingredients train on Speech Accessibility Project data and synthetic speech derived from it, and on nothing from outside the challenge.

What is unusual about this soup

Every other soup in this family averages checkpoints from different training runs — different corpora, or a different augmentation recipe. This one averages five consecutive epochs of a single run, which is a weaker proposition: the ingredients share an optimisation trajectory and agree almost everywhere, so there is less disagreement for averaging to cancel. The expectation going in was a small gain or none.

The measurement came out at the better end of that, and the reason to believe it is the ingredients' own ranking: epoch 9 was the best single checkpoint at 6.33% CER, epoch 10 had already turned back up at 6.36%, and the average of the five beats the best of the five. That is the signature of averaging cancelling noise rather than of picking a lucky checkpoint.

It also costs nothing to produce. The five ingredients were written by the training run itself; no second run, and no second allocation, was needed.

Usage

import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT

model_id = "dys-asr/parakeet-rnnt-0.6b-all-syn-chunk-cutout-soup"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()

audio, rate = sf.read("utterance.wav", dtype="float32")
inputs = processor(audio, sampling_rate=rate, return_tensors="pt").to(device)
with torch.inference_mode():
    generated = model.generate(**inputs)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Requires transformers>=5.9. Audio must be 16 kHz mono.

Ingredients

Uniform weights, 1/5 each. No greedy selection: with ingredients this close, admitting them one at a time by score would be selecting on the same set the result is reported on.

Float tensors are averaged in float64 and cast back. BatchNorm's int64 num_batches_tracked is carried across as a maximum rather than averaged, since a counter has no meaningful mean and integer division would silently floor it. This changes nothing at inference: that counter is read only when a BatchNorm layer sets momentum=None, and Parakeet's layers set a momentum.

The soup is built and saved on CPU, deliberately. cuDNN flattens the prediction network's LSTM weights into one buffer the first time it runs on CUDA, which leaves decoder.lstm.weight_ih_l0 and its siblings sharing storage; safetensors writes that without complaint and then refuses to read it back.

Training data

The ingredients' training set, unchanged — 468,435 records and 1,086.5 hours before filtering, from SAPC1 train and dev, SAPC2 train, 103.1 hours of synthetic CosyVoice speech, and 78.8 hours of chunks force-aligned out of recordings longer than the 45-second training cap. The ingredient card carries the full breakdown, the chunking method, and the augmentation recipe, including the SpecCutout this run was built to test.

Evaluation

Greedy decoding, no language model, the project's transcript normaliser on references and hypotheses alike. All 17,582 utterances from the 48 SAPC2 dev speakers who appear nowhere in the ingredients' training data.

The middle row is 0.13 WER points worse here than the 10.08% on its own card. Both numbers are that checkpoint on this same set; they differ because the training loop evaluates under a bf16 autocast while the standalone scorer loads in float32. The two rows compared above went through the same scorer, which is the comparison the 0.27-point margin rests on; mixing a number from one harness with a number from the other would not be meaningful.

What this does not establish

• No paired significance test. On ~141,000 reference words the standard error of one such WER is roughly 0.08 points, so 0.30 is a few of them — but word errors correlate within an utterance, which widens the true interval.
• Single run, single seed. Averaging five epochs of one trajectory is not the same experiment as averaging five seeds, and says nothing about the latter.
• The evaluation set has no Parkinson's speech. All 35 of SAPC2 dev's Parkinson's speakers also appear in SAPC1 dev, which the ingredients train on, so this set is harder than SAPC2 dev as a whole and is not representative of every etiology in training. Do not compare these numbers with results reported on the full dev split.

Intended use and limitations

Research on atypical-speech recognition, and as a starting point for speaker-specific adaptation.

• Requires transformers>=5.9. Not loadable by the 4.x line.
• Slow relative to CTC. Decoding is autoregressive.
• Lower-case, unpunctuated, numerals written as words, inherited from the ingredients, and unlike the CTC models in this family.
• Inherits every limitation of the ingredient checkpoint, including that its margin over its own predecessor confounds four simultaneous changes. That card states them.
• English only, 16 kHz mono.
• Not a clinical tool. Nothing here supports inference about any diagnosis.

License and attribution

Released under CC BY 4.0, following the upstream NVIDIA Parakeet family. The Speech Accessibility Project corpora are governed by their own data use agreement and are not redistributed here. Fun-CosyVoice3's terms apply to the synthesis component of the ingredients' training data.