DuplexJev-4B-Stress
📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Research note: teaching a speech-decision LLM to hear stress
DuplexJev-4B-Turn that also hears stress and pauses, for smaller GPUs. "I didn't say she took the money" and "I didn't say she took the money" are the same words; this model tells them apart. It answers which word is stressed, is any word stressed at all, where the speaker pauses, and what the stress implies, in Mandarin and English.
Trade-off — read before choosing. At 4B, adding stress costs some spoken-QA accuracy: VoiceBench OBQA / MMSU is 42.0 / 38.2 against 46.4 / 41.7 for 4B-Turn, and qa100 / ZJU-ML are 1 point lower. We tried lower stress weights, detection-only data, fewer steps and up to 4× more content distillation; none closed the gap at 4B (the same recipe costs nothing at 32B). If you do not need stress or pauses, use DuplexJev-4B-Turn; if you have an 80 GB GPU, use DuplexJev-32B-Stress, which has no such trade-off.
DuplexJev reads typed, closed-set decisions from speech as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (Qwen3-ASR-0.6B encoder
- connector + Qwen3-4B, 4.2 B parameters, bf16; ~10 GB GPU memory).
What changed
Starting from DuplexJev-4B-Turn we trained the connector and a new rank-16 LoRA on the LLM (merged here) for 4,000 steps. The training mix is the 4B-Turn mix with its content distillation weighted 2.5×, plus about 23 k prosody questions (which word is stressed, where the pause is, and what the stress or pause means), balanced so that every speech source has both stressed and neutral clips:
| source | lang | what it teaches |
|---|---|---|
| DEBATE (real, 10 readers), training sentences only | zh | which span is stressed; where the pause is |
| Stress-17K (WhiStress-consistent part), TinyStress-15K | en | which word is stressed (synthetic speech) |
| Expresso emphasis style (real), StressPresso clips removed | en | which word is stressed (real speech) |
| CosyVoice2 minimal sets that passed a text check and an acoustic check | zh | which word is stressed (synthetic) |
| AISHELL-1 train, LibriSpeech, Expresso default style, CosyVoice2 neutral readings | zh / en | no word is stressed (negatives) |
Unlike the 32B release, this model also keeps the meaning-choice questions: at 4B, dropping them did not bring VoiceBench back (detection-only: 42.4 / 38.1) but cost 12 points on meaning-from-stress, so the full mix is the better trade-off here. Details and all ablations in the research note.
Results
| DuplexJev-4B-Turn | DuplexJev-4B-Stress | |
|---|---|---|
| qa100 / ZJU-ML | 74 / 52 | 73 / 51 |
| gender / emotion | 89.4 / 92.0 | 89.1 / 91.4 |
| VoiceBench OBQA / MMSU | 46.4 / 41.7 | 42.0 / 38.2 |
| Easy-Turn* / CoDeTT zh / en / TurnBench-dev clips† | 92.5 / 67.0 / 68.6 / 87.3 | 91.8 / 67.0 / 68.9 / 86.9 |
| which word is stressed, "none" offered: DEBATE (zh) / StressPresso (en) | 39 / 26 | 95 / 95 |
| neutral speech → "no word is stressed": AISHELL-1 / LibriSpeech | 78 / 37 | 99 / 99 |
| meaning from stress: DEBATE held-out sentences / StressTest | 36 / 49 | 94 / 86 |
| meaning from pause: DEBATE held-out sentences | 67 | 85 |
Paper protocol (single-token readout, PyTorch), % correct; DEBATE numbers are on sentences that were never in
training. Through vLLM with duplexjev-vllm in the client
wording: qa100 70, ZJU-ML 46, gender 87.5, emotion 90.1, Easy-Turn 91.6 (DuplexJev-4B-Turn in the same run: 77, 49, 87.1,
88.6, 92.9; qa100 and ZJU-ML have 100 items each).
- * Easy-Turn is in-domain (its training split is in the mix). † Our clip-level protocol on the TurnBench development conversations, not comparable with the official leaderboard. CoDeTT is zero-shot, current utterance only.
Serve with vLLM
pip install "vllm[audio]>=0.29" duplexjev-vllm
vllm serve adventists-ai/DuplexJev-4B-Stress --max-model-len 4096
Prompt format and client are the same as for DuplexJev-4B-Turn;
see examples/vllm_client.py.
A stress question, for example:
用户说：<|audio|>
问题：说话人特别强调（重读）的是哪个词？
选项：
A. 我
B. 明天
C. 北京
D. 没有特别强调哪个词
只回答正确选项的字母。
Always offer a "no word is particularly emphasized" option: the model was trained to use it, and without it the question presumes that something is stressed. If you need fewer misses and can accept more false alarms, lower the "none" option's log-probability by a constant before taking the argmax.
Limitations
- Mandarin stress training data is read speech from 10 speakers; stress in spontaneous conversation is weaker and more varied, and has not been evaluated.
- Spoken multiple-choice QA is about 4 points below DuplexJev-4B-Turn (see the trade-off above).
- Decides on one clip at a time; pauses inside long turns are still often taken as the end of the turn.
- Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
License
CC BY-NC 4.0, non-commercial use only. The stress data is CC BY-NC 4.0: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl), Expresso (Meta). The emotion data (ESD) is research-only and SmoothConv is CC BY-NC. Base models keep their licences: Qwen3-4B and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud); the Qwen3-4B licence is included. The LLM weights are Qwen3-4B with LoRAs merged in.
Citation
@misc{jin2026duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
year = {2026},
eprint = {2610.02638},
archivePrefix = {arXiv},
note = {Submitted to IEEE ICASSP 2027}
}
Built by Adventists.ai. Claude (Anthropic) assisted with code.