DuplexJev-32B-Para (v1.1)

📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Research notes: hearing stress · keeping a speech-decision LLM able to talk

The DuplexJev model that hears the most beyond the words. On top of everything DuplexJev-32B-Turn and DuplexJev-32B-Stress do (spoken QA, gender, emotion, turn-taking, stress and pauses), it also hears laughter, breathing, coughs and sighs, notices how someone is talking (whispering, shouting, very fast or slow), and can tell, roughly as a person would, whether two clips are the same speaker. Mandarin and English. Since v1.1 it also holds a normal conversation, multi-turn included.

v1.1 (2026-10-10) replaces v1.0 (2026-10-09) and is better on almost every decision benchmark (table below). v1.0 stays available as revision v1.0 (--revision v1.0 in vLLM, revision="v1.0" in from_pretrained).

DuplexJev reads typed, closed-set decisions from speech as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (Qwen3-ASR-0.6B encoder

• connector + the language model of Qwen3-VL-32B, 33.0 B parameters, bf16; one 80 GB GPU).

Which 32B to pick. This one if you want the widest set of paralinguistic signals. Turn if every point of emotion and turn-taking matters; Stress if stress is the main thing you need (see the table).

What changed

Starting from DuplexJev-32B-Turn we trained the connector and a new rank-16 LoRA on the LLM (merged here) for 4,000 steps. The mix is the 32B-Turn mix plus:

Results

Paper protocol (single-token readout, PyTorch), % correct unless marked; AUC 0.5 = chance. Through vLLM with duplexjev-vllm in the client wording, v1.1: qa100 97, ZJU-ML 91, gender 93.2, emotion 90.0, Easy-Turn 94.8 (v1.0: 95, 90, 91.9, 89.5, 94.1; qa100 and ZJU-ML have 100 items each).

• * Easy-Turn is in-domain (its training split is in the mix).
• ‡ OpenBookQA's train split is in the replay data, so the OBQA gain is partly in-domain; MMSU is unrelated to it.
• For reference, an AudioSet AST tagger scores 0.71 / 0.75 / 0.85 / 0.76 on the same four sounds.
• Speaker: a dedicated speaker-verification model reaches ~1% EER on VoxCeleb1; this model is meant to have a rough, human-like sense of "same voice or not", not to identify people.

Serve with vLLM

pip install "vllm[audio]>=0.29" duplexjev-vllm
vllm serve adventists-ai/DuplexJev-32B-Para --max-model-len 4096

Prompt format and client are the same as for DuplexJev-32B-Turn; see examples/vllm_client.py. Examples:

The user said: <|audio|>

Is there laughter in this audio?

Options:
A. Yes
B. No

Answer with the letter only.

用户说：<|audio|>

除了说话，这段音频里还有哪种声音？

选项：
A. 叹气
B. 咳嗽
C. 笑声
D. 都没有

请只回答选项字母。

Always offer a "none" option for stress and non-verbal questions; the model was trained with it.

Limitations

• Non-verbal training audio is mostly acted (radio drama, animation, studio recordings); in natural conversation the scores are lower (see the table).
• Breathing in natural Mandarin conversation is lower than in v1.0 (AUC 0.65 vs 0.78); acted / read speech is higher.
• Asked about itself, it may describe abilities (seeing images, speaking) that this checkpoint does not have on its own: it takes speech and text and replies in text.
• Speaker judgments are coarse (EER 18–40%) and must not be used to identify or verify people.
• Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.

License

CC BY-NC 4.0, non-commercial use only.

• Non-commercial data:
  • Stress: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl);
  • Expresso (Meta), EARS (Meta) and NonVerbalSpeech-38K, all CC BY-NC 4.0;
  • ESD (research only); SmoothConv (CC BY-NC).
  • Conversation replay (v1.1): user turns from wilzzzz/paralinguistic_dialogues (no licence stated on the dataset card; used for non-commercial research), SmoothConv (CC BY-NC), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0); target replies generated by the base LLM (Apache-2.0); spoken chat questions synthesised with CosyVoice2 (Apache-2.0).
• Other data:
  • VocalSound (CC BY-SA 4.0), NonverbalTTS (audio from VoxCeleb and Expresso);
  • VoxCeleb1 (CC BY 4.0), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0);
  • ARC (CC BY-SA 4.0), OpenBookQA, CommonsenseQA (MIT).
• Base models keep their licences: Qwen3-VL-32B-Instruct and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud).
• The LLM weights are the language model of Qwen3-VL-32B-Instruct with LoRAs merged in; the vision encoder is not included.

Citation

@misc{jin2026duplexjev,
  title         = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author        = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  year          = {2026},
  eprint        = {2610.02638},
  archivePrefix = {arXiv},
  note          = {Submitted to IEEE ICASSP 2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.