Parakeet RNN-T 0.6B — SAPC1 + SAPC2 + synthetic speech
This is an English automatic speech recognition model adapted for dysarthric
speech. It fine-tunes the Transformers-format
extraordinarylab/parakeet-unified-en-0.6b
RNN-T checkpoint on the deduplicated Speech Accessibility Project Challenge
(SAPC) 1 and 2 training recordings plus 103.1 hours of synthetic dysarthric
speech.
The checkpoint is compatible with ParakeetForRNNT in Transformers 5.15.1.
Usage
import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT
model_id = "dys-asr/parakeet-rnnt-0.6b-sapc12-syn"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()
audio, sample_rate = sf.read("speech.wav", dtype="float32")
if audio.ndim == 2:
audio = audio.mean(axis=1)
if sample_rate != 16_000:
raise ValueError("Resample input audio to 16 kHz before inference")
inputs = processor(audio, sampling_rate=sample_rate, return_tensors="pt").to(device)
with torch.inference_mode():
output = model.generate(**inputs)
text = processor.batch_decode(output.sequences, skip_special_tokens=True)[0]
print(text)
The built-in generate() path is greedy. Our Codabench inference adapter uses
a separate time-synchronous RNN-T beam search with beam size 2; num_beams=2
should not be passed to generate() because the model's RNN-T frame and decoder
cache bookkeeping is designed for one greedy path.
Training data
The real-speech portion combines the SAPC1 and SAPC2 training splits. Duplicate recordings re-released in SAPC2 were removed rather than given twice the weight. The combined manifest contained 372,400 real utterances (832.6 hours) from 972 speakers.
The synthetic portion contains 55,988 utterances (103.1 hours), generated with Fun-CosyVoice3 zero-shot voice cloning. Each target text was paired with a 3–10-second SAP reference recording from a speaker with the same reported etiology. Reference speakers were sampled proportional to the square root of their per-speaker CER under a stock recognizer, softly favouring voices that were harder to recognise without concentrating on only the most severe speech.
Synthetic hours were allocated approximately as follows: cerebral palsy 50%, Down syndrome 30%, stroke 17%, and ALS 3%. The text mix was 30% existing SAP transcripts that were rare within the target etiology, 50% novel SAP-like text, and 20% text targeting common recognition errors. Audio was generated deterministically per utterance and stored as 16 kHz mono PCM. Failed or degenerate generations were excluded. The synthetic audio itself is not distributed in this model repository.
After duration and label-length filtering, 422,451 records were used for training.
Training procedure
- Initial checkpoint:
extraordinarylab/parakeet-unified-en-0.6b - Architecture: 0.6B-parameter FastConformer RNN-T
- Epochs: 10
- Optimizer: AdamW
- Peak learning rate:
1e-4 - Scheduler: tri-stage
- Effective batch size: 32 across 16 GPUs
- Precision: bfloat16
- Encoder layerdrop: 0.05
- Speed perturbation factors: 0.8, 0.9, 1.0, 1.1, 1.2
- SpecAugment: enabled
- Maximum input duration: 45 seconds
- Seed: 42
Evaluation
Greedy decoding on the complete held-out SAPC2 development split produced:
| Metric | Result |
|---|---|
| WER | 11.32% |
| CER | 7.16% |
References and hypotheses were normalized with the project's English text normalizer before scoring. These figures come from the checkpoint-selection evaluation pipeline; no beam-search result is claimed here.
Intended use and limitations
This model is intended for research and evaluation of English ASR for speakers with dysarthria, including the SAP Challenge setting. It may also be useful as an initialization for further speaker or domain adaptation.
Performance varies by speaker, etiology, severity, recording conditions, and prompt type. SAP data is not representative of every person with dysarthria, and synthetic voice-cloned speech may reproduce artifacts or biases of both the reference recordings and the TTS system. The model can omit, substitute, or hallucinate words; it should not be used as the sole basis for clinical, legal, safety-critical, or access decisions. Obtain informed consent before processing identifiable speech and follow the SAP data-use terms when working with the source datasets.
License and attribution
Released under CC BY 4.0, following the upstream NVIDIA Parakeet model family. Please also follow the terms of the SAP datasets and Fun-CosyVoice3 for any separately obtained training data or synthesis components.