TR-Omni-E4B — Turkish native speech-to-speech model

TR-Omni-E4B is a two-component (thinker–talker) Turkish speech-to-speech model. The thinker, Gemma 4 E4B adapted to Turkish speech, takes speech, images and text as input and writes the reply; speech is encoded directly, without an intermediate ASR transcript. The talker, initialised from Trendyol-TTS (VoxCPM2), generates 48 kHz speech one sentence at a time, conditioned on the thinker's sampled tokens together with its hidden states for them. The real-time browser app (sentence-level streaming, turn-taking, barge-in) is on GitHub: Furkansz/TR-Omni-E4B.

Positioning. The input side is native: the thinker encodes speech itself, with no ASR step. The output side follows the thinker–talker design of Qwen2.5-Omni and LLaMA-Omni2: a separate talker, here initialised from a TTS model, reads the reply's tokens together with the thinker's contextual hidden states. TR-Omni-E4B is therefore neither a cascade that hands only text to an independent TTS, nor a single model that generates audio directly (as GPT-4o or Moshi do). Pronunciation comes mainly from the tokens; whether the hidden states improve prosody has not been measured yet (see Limitations).

Türkçe — TR-Omni-E4B, iki bileşenli (thinker–talker) bir Türkçe speech-to-speech modelidir. Thinker (Türkçe konuşmaya uyarlanmış Gemma 4 E4B) konuşma, görüntü ve metin girdilerini alır ve cevabı yazar; konuşma girdisi ara bir ASR transkriptine dönüştürülmeden doğrudan işlenir. Trendyol-TTS'ten (VoxCPM2) başlatılan talker ise 48 kHz konuşmayı cümle cümle üretir ve bunu thinker'ın ürettiği token'larla birlikte bu token'lara ait hidden state'leri kullanarak yapar. Yani girdi tarafı native'dir; çıkış tarafı ise Qwen2.5-Omni ve LLaMA-Omni2'deki thinker–talker yapısını izler. Model, yalnızca metni bağımsız bir TTS'e aktaran bir cascade değildir; GPT-4o veya Moshi gibi sesi doğrudan üreten tek bir model de değildir. Tarayıcıda canlı sesli sohbet uygulaması GitHub'dadır: Furkansz/TR-Omni-E4B.

Highlights

• State-conditioned speech generation. The talker is conditioned on the thinker's tokens and its hidden states (layers 14, 28, 42), fused per talker token by a learned per-dimension gate.
• Sentence-level generation. Replies are voiced one sentence at a time. In the streaming live app, first audio arrives ≈ 0.7 s after the end of the user's turn is detected, on a 2g MIG slice of an H200 (median over 20 spoken questions); the offline tr_omni.py returns the audio after the full reply.
• Turkish ASR. 10.2 % WER on 1,000 Common Voice 17 tr test utterances (base Gemma 4 E4B: 14.8 %); on par with the base on FLEURS (743 utterances).
• Vision retained. The Gemma 4 vision tower is unchanged; on a 40-image synthetic check, 38/40 typed and 37/40 spoken questions were answered correctly.
• Paralinguistic cues. Speaker gender 92.5 % (n = 120) and non-verbal vocal sounds 96.6 % (n = 29), on small diagnostic sets.
• Open weights under Apache-2.0, with minimal PyTorch / Transformers inference code.

Architecture

The thinker generates the reply greedily. Each completed sentence is normalised to its spoken form and re-encoded in context (system prompt, the last 4 s of the user turn and the reply so far), which yields hidden states from layers 14, 28 and 42, i.e. 3 × 2560 per thinker token. Each talker token is aligned to the thinker token containing its first character. A StateMapper (softmax layer mixing, LayerNorm, 2560 → 1024 projection with piece-position embeddings, MLP to 2048) maps the states into the talker's text-embedding space, where a per-dimension gate mixes them with the talker's own token embeddings (mean g ≈ 0.4). The talker, consisting of a text-semantic LM, a residual acoustic LM, a local diffusion transformer and an AudioVAE decoder, continues from a fixed synthetic voice prompt. Token conditioning keeps pronunciation exact; states alone preserve content but not orthography (see the ablation below). This follows the hidden-state and token fusion used by Qwen2.5-Omni and LLaMA-Omni2.

Evaluation

All results are author-run on the released weights with greedy decoding; sample sizes are given per test. Generated speech is scored by transcribing it with Whisper large-v3 and comparing the transcript with the intended text. The baseline is the unmodified Gemma 4 E4B-it given identical inputs.

Speech recognition (WER / CER %)

Identical prompt for both models ("Bu ses kaydını yazıya dök."). Normalisation: Turkish-aware lower-casing (I/ı), numbers spelled out, punctuation removed, apostrophes dropped.

The YODAS row uses a different normalisation and is not directly comparable with the rows above. Without apostrophe removal, FLEURS and Common Voice give 7.2 vs 6.7 and 10.8 vs 15.4; TR-Omni tends to insert apostrophes after common nouns.

Speech generation: talker ablation

100 sentences from the FLEURS tr_tr test split (unseen in training), each generated as the reply to a spoken request ("Bana bir şey anlatır mısın?") with in-context states, as in live use.

Token + state conditioning stays within 0.2 CER points of the TTS it was initialised from; residual errors concentrate on foreign proper nouns ("pohutukawa", "Dunlap") and occasional elided syllables. The ablation shows that the tokens anchor lexical accuracy. It does not show that the states improve intelligibility over text-only TTS; their intended role is to carry conversational context into the talker.

Spoken question answering

116 general-knowledge questions in Turkish (12 topics; 40 easy, 48 medium, 28 hard), synthesised with voices unseen in training. Answers are judged against short references by Gemma-4-31B-it (binary).

The gap on hard questions reflects knowledge loss from speech adaptation. Numbers are verbalised in the reply, which accounts for most arithmetic errors ("On iki çarpı on iki kaç eder?" → "Yüz altmış dört eder."). The "as heard" row also includes evaluator ASR errors ("on beş" → "415"); spoken output agrees with the answer text at 3.2 % WER / 0.9 % CER, and most of the remaining gap comes from symbols read letter by letter ("Au" → "a u", "H2O" → "he iki o").

Vision

40 synthetic images with known answers, 5 per skill (Turkish text reading, counting, highest and lowest bar, table lookup, colour, arithmetic, left/right). Each question is asked typed and spoken, the latter with the live system prompt.

Both models are error-free on text reading, charts, tables, colour and left/right; counting is the shared weakness (7 or 8 red circles among blue squares). The two spoken-question misses in arithmetic are verbalisation errors ("24" → "otuz dört"); the typed answers are correct.

Paralinguistics (accuracy)

Scored by label match; the base model rarely names a label. These are small diagnostic sets: emotion recognition is weak, and the model should not be used as an emotion or speaker-attribute classifier.

Quickstart

Download the full repository (thinker/, talker/, tr_omni.py, tr_speak.py) and run it from its root:

pip install torch "transformers>=5.17" voxcpm==2.0.3 soundfile numpy librosa pillow
python tr_omni.py --audio question.wav --out reply.wav                       # speech → speech
python tr_omni.py --audio question.wav --image photo.jpg --out reply.wav     # speech + image → speech
python tr_omni.py --text "Kapadokya'yı anlatır mısın?" --out reply.wav       # text → speech

from tr_omni import TROmni
omni = TROmni(".")                                # thinker/ and talker/
text, wav = omni.chat(audio=wav16k)               # float32 16 kHz mono in, 48 kHz float32 out
text, wav = omni.chat(audio=wav16k, image=img)    # img: PIL image

history = []                                      # multi-turn: pass the same list every turn
text, wav = omni.chat(audio=turn1, history=history)
text, wav = omni.chat(audio=turn2, history=history)

Live conversation in the browser. The streaming app (sentence-level streaming, Smart Turn v3.2 turn-taking, echo-cancelled barge-in in Chrome) is in a separate repository and downloads this model on its first run:

git clone https://github.com/Furkansz/TR-Omni-E4B.git && cd TR-Omni-E4B
pip install -r requirements.txt
python live/server.py                             # then open http://localhost:7860

Hardware. Measured peak: 15.4 GiB allocated by PyTorch, excluding the CUDA context and allocator overhead, so a 16 GB GPU may not be enough; 24 GB is comfortable. The thinker's 5.6 GB per-layer embedding table is kept in host RAM. Decoding uses a static KV cache with one CUDA graph per token; the first call includes about a minute of compilation. tr_omni.py is non-streaming: a two-sentence reply (9 s of audio) completes 14 s after the query on a 2g MIG slice of an H200. The ≈ 0.7 s first-audio latency refers to the sentence-level streaming live app.

Training

Thinker. Gemma 4 E4B-it, LoRA (r = 64) on the language model in two stages, merged into the released weights.

1. Speech (2,000 steps), with the audio encoder and audio projector also trained: transcription (Common Voice 17, FLEURS, MediaSpeech, YODAS); responses to spoken requests and content questions, self-distilled from Gemma 4 E4B-it on transcripts (as in BLSP and DeSTA2.5); speech translation into English; questions about delivery; paralinguistic questions (VocalSound, CREMA-D); text-only replay (Aya).
2. Omni (2,000 steps, audio encoder frozen): spoken replies prefixed with a tone tag, including synthetic users with 9 emotions (VoxCPM2 voice design; dialogues by Gemma-4-31B-it), emotion recognition, and the stage-1 mix.

The vision tower and vision projector are unchanged.

Talker. Initialised from Trendyol-TTS (a Turkish VoxCPM2) and trained on ≈ 114 k Turkish utterances (YODAS, Common Voice 17, FLEURS train), with LoRA (r = 32) on the attention layers of the text-semantic LM, residual acoustic LM and local diffusion decoder, merged into the released weights.

1. StateMapper pre-training with state-only conditioning (the "states only" row above).
2. Token + state fusion through the learned gate, with modality dropout (both / states / tokens) and same-speaker continuation targets.
3. The same objective with hidden states extracted in conversational context (the preceding utterance as the user turn), on 48 k utterances.

Limitations

• Knowledge. 4.5 B effective parameters; factual and arithmetic errors occur, and hard-question accuracy is below the base model. Retrieval is recommended for factual use.
• Voice and prosody. A single synthetic voice; prosody does not adapt to the conversation. Emotion recognition is weak.
• Role of the hidden states. Measured so far only for intelligibility, where token + state conditioning is close to, but not better than, text-only TTS (3.0 % vs 2.1 % WER). Their effect on prosody and contextual fit has not been evaluated yet.
• Turn-taking. Half duplex. End-of-turn detection, streaming and barge-in are application concerns (the live app uses Smart Turn v3.2 and echo-cancelled barge-in).
• Normalisation. Speech follows tr_speak.py; rare abbreviations, foreign names and uncommon notation may be misread.
• Language. Turkish only.
• High-stakes use. Not intended for medical, legal or financial decisions.

Responsible use

The assistant voice (talker/voice/assistant.wav) is synthetic, generated by Trendyol-TTS without a reference speaker. The talker continues the voice of its prompt and can therefore imitate a voice from a short sample. In line with the VoxCPM2 terms, do not use this model to impersonate real people, for fraud or for disinformation, and label generated speech as AI-generated. Do not clone voices without consent.

License

Apache-2.0 (LICENSE); third-party notices in NOTICE.md. Built on Gemma 4 E4B-it (Apache-2.0), VoxCPM2 (Apache-2.0) and Trendyol-TTS (MIT; trained by its authors on a private Turkish speech dataset). Training data: Mozilla Common Voice 17.0 (CC0-1.0), Google FLEURS (CC-BY-4.0), MediaSpeech (CC-BY-4.0), YODAS (CC-BY-3.0), Aya (Apache-2.0), VocalSound (CC-BY-SA-4.0), CREMA-D (ODbL-1.0), and synthetic text and speech from Gemma 4 and VoxCPM2. Turn detection in the live app: Smart Turn v3 (BSD-2-Clause).