Qwen3 1.7B Omni-token checkpoint 23447
MV text and mv text converted to audio using qwen3-0.6b with 8 voices.
mv text : 110B interleaved (text , audio): ~6.75B Speech → speech : ~6.75B Speech → text continuatio : ~6.75B Text → speech continuation : ~6.75B Direct STT : 5B Direct TTS : 5B
This is iteration 23,447 of a dense Qwen3 1.7B architecture trained with one shared autoregressive vocabulary for text, <listen> input tokens, and <speak> output tokens. The package includes the expanded tokenizer and portable inference code for:
- text to speech (TTS)
- speech to text (STT)
- text to speech continuation
- speech to text continuation
- speech to speech continuation
The checkpoint and scripts contain no audio files. Supply your own input/reference audio at runtime. Audio is encoded and decoded with hubertsiuzdak/snac_24khz, which is downloaded on first use unless already cached.
Install
Use Python 3.11 and install the correct PyTorch build for your CPU or CUDA platform first. The versions in requirements.txt are the tested environment; if the pinned PyTorch wheel is unavailable for your platform, install the matching build from PyTorch and then install the remaining packages.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Generate
All speech outputs are 24 kHz WAV files. Reference audio is used for voice conditioning and may be WAV, FLAC, Opus, or another format supported by SoundFile/FFmpeg.
# TTS in a supplied reference voice
python inference.py --task tts \
--reference-audio /path/to/reference.flac \
--text "The model is speaking this sentence." \
--output generated.wav
# STT
python inference.py --task stt \
--audio /path/to/input.opus \
--output transcript.txt
# Continue text as speech in a supplied reference voice
python inference.py --task text-speech-continuation \
--text "The first part of the document ends here." \
--reference-audio /path/to/reference.flac \
--output continuation.wav
# Continue speech as text
python inference.py --task speech-text-continuation \
--audio /path/to/excerpt.opus \
--output continuation.txt
# Continue speech as speech
python inference.py --task speech-speech-continuation \
--audio /path/to/excerpt.opus \
--output continuation.wav
When the files are downloaded from Hugging Face into another directory, pass either that local directory or the repository ID:
python inference.py --model ORG/REPOSITORY --task tts \
--reference-audio reference.flac --text "Hello." --output hello.wav
Useful controls include --greedy, --temperature, --top-p, --repetition-penalty, --max-new-tokens, --device, and --dtype. --local-files-only prevents network access once both this model and the SNAC codec are cached.
Media-token contract
Input audio uses SNAC levels 0 and 1 packed as three tokens per base frame inside <listen>...</listen>. Generated speech uses all seven SNAC code streams per base frame inside <speak>...</speak>. The tokenizer validates these reserved token IDs before loading the model, preventing accidental use with an incompatible tokenizer.
Limitations
This is a research checkpoint, not a production speech system. TTS has produced audible outputs in project testing. STT and cross-modal continuation quality remain experimental and can be poor, repetitive, or silent for some prompts and references. Generated speech should be reviewed before use.