Direct Speech-to-Speech Translation: kaz → {tur, tat, uzb}
Trained checkpoints for a direct (no intermediate text at inference time) speech-to-speech translation system, Kazakh → Turkish / Tatar / Uzbek. Code: github.com/NLP-KazNU/s2s-public. Paper: IEEE Access, in revision (citation to be added on acceptance).
Architecture and checkpoints
| File | Stage | Size |
|---|---|---|
whisper_s2tt_ws2tt_lv3.pt | S2TT text stage: fine-tuned Whisper large-v3 encoder + custom FiLM decoder (whisper_s2tt.py) | 2.7 GB |
codec_decoder_l9_head.pt | Single-stream S2UT unit predictor: CodecDecoderXAR, n_q=1, mHuBERT-147 layer-9 units (codec_decoder_xattn.py) | 122 MB |
unit_vocoder_hifigan.pt | Dedicated Turkic single-stream unit vocoder (train_unit_voc_hifigan.py) | 338 MB |
Loading code and configs live in the GitHub repo;
these .pt files are raw state_dict checkpoints matching the model classes
defined there (WhisperS2TT, CodecDecoderXAR, SpeechBrain UnitHifiganGenerator).
Results (test set)
- S2TT text-BLEU avg 19.05 (tur 12.76 / tat 25.76 / uzb 18.63)
- End-to-end ASR-BLEU (argmax) avg 3.25 (tur 1.30 / tat 3.22 / uzb 5.24)
- Beats an MMS→MADLAD→MMS cascade baseline (1.56 avg) by >2×
Full numbers, ablations, and the honest metric pack (chrF++/WER/CER/COMET/SQUIM/RTF)
are in RESULTS.md.
Audio samples
See the demo_audio/ folder — 8 example system outputs (kaz source →
predicted tur/tat/uzb speech) from the final single-stream S2UT system.
What this is not
Earlier architecture iterations (from-scratch encoder with morphology/VQ/SONAR auxiliary losses, a non-autoregressive mel decoder, an EnCodec-based acoustic-code predictor) all produced ASR-BLEU ≈ 0 and are not published — see the paper's Limitations section and the GitHub repo's README for why.
License
MIT.