Neurlang Whipstr STT (ASR)
A deep learning automatic speech recognition (ASR) system for transcribing speech audio into IPA text using transformer-based sequence-to-sequence models.
- Language: Universal (IPA), 74+ languages
- Model Github: neurlang/whipstr https://github.com/neurlang/whipstr
- Model Dataset: Common Voice 21
- Model-Native Sample Rates: 8000 Hz, 16000 Hz, 24000 Hz, 32000 Hz, 48000 Hz
- Degraded-Performance Sample Rates: 11025 Hz, 22050 Hz, 44100 Hz
- License: GPL v2
- Total parameters:
- Encoder: 7 253 472
- Transformer: 99 828 768
- Total (trainable): 107 082 240
- Total (incl. positional-encoding buffers, as stored in safetensors): ~114.8M
- CER: 24.89% (75.11% success rate)
- Note: Averaged across all supported languages, works better on higher resource languages
- WER: 60.25% (39.75% success rate)
- Note: Averaged across all supported languages, works better on higher resource languages
Architecture
encoder_embed_dim=192, d_model=768, nhead=12, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=3072, dropout=0.15, stride=1, window_size=11- ~2.07× whipstr-medium's 51.7M params, ~7.26× whipstr-base's 14.76M
- In Whisper terms: ~107M lands just above Whisper-base (74M), well below Whisper-small (244M)
- Note: The CNN encoder (
conv1–conv4/pool1–pool3/fc1) is hardcoded regardless of variant — only the finalfc2bottleneck layer scales withoutput_values(7.22M→7.24M→7.25M base→medium→large). All size scaling across variants is in the transformer, not the acoustic front-end. (Documented upstream; seewhipstr_encoder.py.)
Inference code
git clone https://github.com/neurlang/whipstr.git
cd whipstr/
uv run --with torch --with transformers --with phase-spectrogram stt_infer_hf.py --audio /home/m/Downloads/LJ001-0001.wav --model neurlang/ipa-whipstr-large-48khz-cv-21
Inference code (microphone)
uv run --with torch --with transformers --with phase-spectrogram --with sounddevice stt_mic_hf.py --model neurlang/ipa-whipstr-large-48khz-cv-21
Training history
Training has run across multiple sessions, with a mid-run switch from batch_size=1 to batch_size=4 (alongside AMP/mixed-precision and other throughput fixes) and several recoveries from unrelated infrastructure issues (a training-restart bug that silently dropped the encoder's weights, and a disk-full interruption) — none of which reflect problems with the model itself, and all of which the checkpoint history below has been verified to recover cleanly from.
Cumulative compute to date
| Run | Batch size | Duration | Batches | Samples |
|---|---|---|---|---|
| run1 | 1 | 2h13m | 45,000 | 45,000 |
| run2 | 1 | 57h02m | 1,158,000 | 1,158,000 |
| run3 | 4 | 4h12m | 20,000 | 80,000 |
| run4 | 4 | 13h22m | 120,000 | 480,000 |
| run5 | 4 | 38h36m | 341,000 | 1,364,000 |
| run6 | 4 | 41h48m | 356,000 | 1,424,000 |
| Total | 157h13m (~6.55 days) | 2,040,000 | 4,551,000 |
Note: because the training script does not yet persist a global step counter across restarts (only epoch), batch numbers within runs 3–6 are restart-local, not a single continuous count — the table above reports each run's own step count and the wall-clock/sample totals, which are unaffected by that.
Key checkpoints (batch_size=1 phase, original EOS/padding scheme)
| Step | WER | CER | Notes |
|---|---|---|---|
| 152,000 | 155.62% | 90.17% | Early/noisy phase |
| 182,000 | 133.84% | 86.27% | First real improvement |
| 359,000 | 128.97% | 85.15% | Plateau |
| 507,000 | 110.89% | 69.00% | Breakout |
| 632,000 | 99.30% | 59.48% | Transient language-misidentification failure mode observed (resolved by step 1,071,000) |
| 1,048,000 | 81.46% | 38.83% | Large jump |
| 1,071,000 | 79.84% | 38.80% | Misidentification pattern confirmed resolved |
| 1,158,000 | — | — | Last checkpoint before switch to batch_size=4 + EOS/padding-mask fixes |
Key checkpoints (batch_size=4 phase, corrected EOS token + padding mask)
| Step (restart-local) | WER | CER | Notes |
|---|---|---|---|
| 15,000 | 79.72% | 41.16% | Recovered from regime-switch dip |
| 49,000 | 73.00% | 33.24% | First checkpoint to cross ipa-whisper-medium's 33.88% CER benchmark |
| 75,000 | 73.00% | 33.31% | Plateau |
| 179,000 | 71.84% | 31.72% | Breakout begins |
| 203,000 | 67.67% | 29.81% | First sub-30% CER |
| 356,000 | 60.25% | 24.89% | Current best (run6) |
Known limitations
- Best performance is on higher-resource languages; CER/WER are averaged across all 74+ supported languages and vary substantially by language
- A small number of persistently hard utterances (very short references, rare-language samples) contribute disproportionately to the current error rate; typical/common-language utterances already perform noticeably better than the aggregate CER suggests