GLM-4-Voice of Reason

A speech-to-speech model that reasons before it speaks. GLM-4-Voice-9B trained with reinforcement learning against a binary LLM judge on math word problems — no supervised finetuning stage, the reasoning behaviour is learned from the reward alone.

Scores 0.706 on GSM8K (1310 test items, written channel, gpt-4o-2024-11-20 as judge).

Run it on a wav

The audio front end is GLM-4-Voice's, unchanged and not in this repo, so clone it for the speech tokenizer:

git clone https://github.com/THUDM/GLM-4-Voice
uv init glm-of-reason && cd glm-of-reason
uv add "transformers>=4.44,<4.48" torch torchaudio accelerate tiktoken soundfile

tiktoken is needed by the tokenizer's remote code and soundfile by torchaudio to read your file; both fail late and unhelpfully if missing. Save the script below as demo.py, point AUDIO at your wav — anything torchaudio can read, any sample rate — and run uv run demo.py:

import sys, torch
sys.path.insert(0, "../GLM-4-Voice")           # the clone, next to the project

from transformers import AutoModel, AutoTokenizer, WhisperFeatureExtractor
from speech_tokenizer.modeling_whisper import WhisperVQEncoder
from speech_tokenizer.utils import extract_speech_token

AUDIO = "question.wav"                                    # <-- your spoken question
REPO = "kyutai/glm-4-voice-of-reason-9b"

SYSTEM = (
    "User will provide you with a speech instruction. Do it step by step. "
    "First, think about the instruction and respond in a interleaved manner, "
    "with 13 text token followed by 26 audio tokens. "
)

tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
model = AutoModel.from_pretrained(
    REPO, torch_dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
).eval()

whisper = WhisperVQEncoder.from_pretrained("THUDM/glm-4-voice-tokenizer").eval().to("cuda")
features = WhisperFeatureExtractor.from_pretrained("THUDM/glm-4-voice-tokenizer")

audio_tokens = extract_speech_token(whisper, features, [AUDIO])[0]
user = "<|begin_of_audio|>" + "".join(f"<|audio_{t}|>" for t in audio_tokens) + "<|end_of_audio|>"
prompt = f"<|system|>\n{SYSTEM}<|user|>\n{user}<|assistant|>streaming_transcription\n"

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(
    **inputs,
    max_new_tokens=1000,
    do_sample=False,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.convert_tokens_to_ids("<|user|>"),
)
new = out[0, inputs.input_ids.shape[1]:].tolist()

audio_offset = tokenizer.convert_tokens_to_ids("<|audio_0|>")
text = tokenizer.decode([t for t in new if t < audio_offset], skip_special_tokens=True)
speech_tokens = [t - audio_offset for t in new if t >= audio_offset]

print(text)

There is no hidden channel here: the model reasons in the answer it speaks, step by step, so text is the whole response. Its stitch sibling keeps the reasoning unspoken instead, wrapped in [SOPR] ... [EOPR].

speech_tokens are the audio codes of that answer; feed them to THUDM/glm-4-voice-decoder to get a waveform, or run the full duplex demo from the GLM-4-Voice repo with --model-path pointing here.

This script was run as printed, on one H100, with transformers 4.47.1 and torch 2.8.0. Pick a torch build that matches your driver: the newest wheel needs a newer CUDA than many clusters run.

License

Inherited from GLM-4-Voice; see the license link above.