DuplexJev-A · gender + emotion, mixed objective (research only)

DuplexJev reads typed, closed-set decisions from speech (is the turn over? which filler fits? who is speaking?) as a single-token distribution of a frozen LLM, with zero decode steps and many questions per forward pass. This repository holds one connector that joins a frozen ASR encoder to a frozen LLM, in Ultravox format.

Quick start

pip install "duplexjev[speech]>=0.2.1"

from duplexjev import Decider, Question

d = Decider.from_pretrained("adventists-ai/DuplexJev-A-Para-Qwen3-ASR-0.6B-Qwen3-32B", device="auto")  # Qwen3-32B bf16: ~65 GB of GPU memory

# English clip
d.decide("utterance_en.wav", [
    Question("gender", "What is the perceived gender of the speaker?", ["female", "male"], lang="en"),
    Question("emotion", "What is the speaker's emotional state?", ["neutral", "happy", "angry", "sad"], lang="en"),
], lang="en")
# Chinese clip: ask in Chinese, with Chinese options
d.decide("utterance_zh.wav", [
    Question("gender", "说话人的性别是？", ["男性", "女性"], lang="zh"),
    Question("emotion", "说话人当时的情绪状态是？", ["中性", "高兴", "生气", "伤心"], lang="zh"),
], lang="zh")
# -> {question_id: {"answer": ..., "confidence": ..., "probs": {option: p, ...}}}

Ask in the language of the clip, with options worded as in the examples: the connector was trained with language-matched prompts. Use duplexjev>=0.2.1; 0.2.0 padded audio at the start, which costs several points. Already have local copies of the encoder and LLM? Pass text_model="/path/Qwen3-32B" and audio_model="/path/Qwen3-ASR-0.6B-Encoder-XAttn"; this works offline.

decide_batch answers question groups over many clips in one pass. See the GitHub repository for the batched pipeline, prefix sharing, the serving loop and evaluation scripts.

Results (paper, Tables 2–3)

qa100: 100 bilingual spoken multiple-choice questions (adventists-ai/qa100). ZJU-ML: main-language part of the ZJU audio benchmark v2.0.0 (50 real-speech factual + 50 TTS math/logic items). Numbers are single-token readouts with the paper's evaluation prompts.

Reproduced with duplexjev 0.2.1 and its default prompts (one option order, lang matched to the clip):

Training

• Frozen: Qwen3-ASR-0.6B encoder and Qwen3-32B. Trained: the connector only.
• Content rounds (R1–R2): transcript distillation (token-level KL to the LLM's transcript-conditioned output, T = 2).
• Decision training: answer-token supervision, cross-entropy on the single option letter at the readout position. Transcript distillation alone cannot teach speaker cues, because its teacher never hears the voice.
• Data: R2 data + gender pack (AISHELL-1, LibriSpeech) + emotion pack (ESD, CREMA-D).

Limitations

• Evaluated on read or acted speech and small test sets (100–800 items); not on streaming input.
• Paralinguistic training lowers accuracy on real spontaneous-style factual questions (see ZJU-ML above).
• Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
• The connector only works with the encoder and LLM listed above.

License

CC BY-NC 4.0, research use only. This connector was trained on the ESD emotional speech corpus, which is licensed for research purposes only, and on CREMA-D (ODbL). Do not use it for commercial purposes. The frozen base models keep their own licences (Qwen3-32B and Qwen3-ASR-0.6B: Apache-2.0).

Citation

@inproceedings{jin2027duplexjev,
  title     = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author    = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  booktitle = {Submitted to IEEE ICASSP},
  year      = {2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.