DuplexJev-32B-Stress
📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Research note: teaching a speech-decision LLM to hear stress
DuplexJev-32B-Turn that also hears stress and pauses. "I didn't say she took the money" and "I didn't say she took the money" are the same words. This model can tell them apart: it answers which word is stressed, is any word stressed at all, and where the speaker pauses, in Mandarin and English. Spoken QA, gender, emotion and turn-taking stay at the 32B-Turn level. Same size, same serving stack, same prompt format as the other DuplexJev models.
DuplexJev reads typed, closed-set decisions from speech as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (Qwen3-ASR-0.6B encoder
- connector + the language model of Qwen3-VL-32B, 33.0 B parameters, bf16; one 80 GB GPU).
What changed
Starting from DuplexJev-32B-Turn we trained the connector and a new rank-16 LoRA on the LLM (merged here) for 4,000 steps. The training mix is the 32B-Turn mix (content distillation, gender, emotion, turn-taking) plus about 14 k prosody detection questions, balanced so that every speech source has both stressed and neutral clips:
| source | lang | what it teaches |
|---|---|---|
| DEBATE (real, 10 readers), training sentences only | zh | which span is stressed; where the pause is |
| Stress-17K (WhiStress-consistent part), TinyStress-15K | en | which word is stressed (synthetic speech) |
| Expresso emphasis style (real), StressPresso clips removed | en | which word is stressed (real speech) |
| CosyVoice2 minimal sets that passed a text check and an acoustic check | zh | which word is stressed (synthetic) |
| AISHELL-1 train, LibriSpeech, Expresso default style, CosyVoice2 neutral readings | zh / en | no word is stressed (negatives) |
Questions that ask for the meaning of a stressed sentence (same words, choose by prosody) are left out on purpose. They raised meaning scores further but cost 2–4 points on spoken multiple-choice QA (VoiceBench); see the research note.
Results
| DuplexJev-32B-Turn | DuplexJev-32B-Stress | |
|---|---|---|
| qa100 / ZJU-ML | 96 / 87 | 96 / 87 |
| gender / emotion | 91.5 / 91.1 | 92.1 / 91.4 |
| VoiceBench OBQA / MMSU | 85.5 / 72.1 | 86.2 / 71.4 |
| Easy-Turn* / CoDeTT zh / en / TurnBench-dev clips† | 95.2 / 69.2 / 70.0 / 88.7 | 94.4 / 69.7 / 70.5 / 88.3 |
| which word is stressed, "none" offered: DEBATE (zh) / StressPresso (en) | 35 / 1 | 92 / 88 |
| neutral speech → "no word is stressed": AISHELL-1 / LibriSpeech | 92 / 50 | 99 / 99 |
| meaning from stress: DEBATE held-out sentences / StressTest | 35 / 52 | 83 / 67 |
| meaning from pause: DEBATE held-out sentences | 71 | 75 |
Paper protocol (single-token readout, PyTorch), % correct; DEBATE numbers are on sentences that were never in
training. Through vLLM with duplexjev-vllm in the client
wording: qa100 97, ZJU-ML 86, gender 93.4, emotion 90.2, Easy-Turn 94.1 (DuplexJev-32B-Turn in the same run: 97, 90, 92.4,
90.5, 95.6; qa100 and ZJU-ML have 100 items each).
- * Easy-Turn is in-domain (its training split is in the mix). † Our clip-level protocol on the TurnBench development conversations, not comparable with the official leaderboard. CoDeTT is zero-shot, current utterance only.
- Differences on the first four rows are within evaluation noise.
Serve with vLLM
pip install "vllm[audio]>=0.29" duplexjev-vllm
vllm serve adventists-ai/DuplexJev-32B-Stress --max-model-len 4096
Prompt format and client are the same as for DuplexJev-32B-Turn;
see examples/vllm_client.py.
A stress question, for example:
用户说：<|audio|>
问题：说话人特别强调（重读）的是哪个词？
选项：
A. 我
B. 明天
C. 北京
D. 没有特别强调哪个词
只回答正确选项的字母。
Always offer a "no word is particularly emphasized" option: the model was trained to use it, and without it the question presumes that something is stressed. If you need fewer misses and can accept more false alarms, lower the "none" option's log-probability by a constant before taking the argmax.
Limitations
- Mandarin stress training data is read speech from 10 speakers; stress in spontaneous conversation is weaker and more varied, and has not been evaluated.
- Choosing the meaning from stress is weaker than detecting the stressed word (83 vs 92 on DEBATE).
- Decides on one clip at a time; pauses inside long turns are still often taken as the end of the turn.
- Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
License
CC BY-NC 4.0, non-commercial use only. The stress data is CC BY-NC 4.0: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl), Expresso (Meta). The emotion data (ESD) is research-only and SmoothConv is CC BY-NC. Base models keep their licences: Qwen3-VL-32B-Instruct and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud). The LLM weights are the language model of Qwen3-VL-32B-Instruct with LoRAs merged in; the vision encoder is not included.
Citation
@misc{jin2026duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
year = {2026},
eprint = {2610.02638},
archivePrefix = {arXiv},
note = {Submitted to IEEE ICASSP 2027}
}
Built by Adventists.ai. Claude (Anthropic) assisted with code.