DuplexJev-A · content (R2)

DuplexJev reads typed, closed-set decisions from speech (is the turn over? which filler fits? who is speaking?) as a single-token distribution of a frozen LLM, with zero decode steps and many questions per forward pass. This repository holds one connector that joins a frozen ASR encoder to a frozen LLM, in Ultravox format.

Quick start

pip install "duplexjev[speech]>=0.2.1"

from duplexjev import Decider, Question

d = Decider.from_pretrained("adventists-ai/DuplexJev-A-Qwen3-ASR-0.6B-Qwen3-32B", device="auto")  # Qwen3-32B bf16: ~65 GB of GPU memory

# English clip
d.decide("utterance_en.wav", [
    Question("turn", "Has the user finished speaking?", ["finished", "not finished"], lang="en"),
    Question("intent", "What does the user want?", ["climate", "media", "navigation", "phone", "chit-chat"], lang="en"),
], lang="en")
# Chinese clip: ask in Chinese, with Chinese options
d.decide("utterance_zh.wav", [
    Question("turn", "用户说完了吗？", ["说完了", "还没说完"], lang="zh"),
    Question("intent", "用户想做什么？", ["空调", "媒体", "导航", "电话", "闲聊"], lang="zh"),
], lang="zh")
# -> {question_id: {"answer": ..., "confidence": ..., "probs": {option: p, ...}}}

Ask in the language of the clip, with options worded as in the examples: the connector was trained with language-matched prompts. Use duplexjev>=0.2.1; 0.2.0 padded audio at the start, which costs several points. Already have local copies of the encoder and LLM? Pass text_model="/path/Qwen3-32B" and audio_model="/path/Qwen3-ASR-0.6B-Encoder-XAttn"; this works offline.

decide_batch answers question groups over many clips in one pass. See the GitHub repository for the batched pipeline, prefix sharing, the serving loop and evaluation scripts.

Results (paper, Tables 2–3)

qa100: 100 bilingual spoken multiple-choice questions (adventists-ai/qa100). ZJU-ML: main-language part of the ZJU audio benchmark v2.0.0 (50 real-speech factual + 50 TTS math/logic items). Numbers are single-token readouts with the paper's evaluation prompts.

Reproduced with duplexjev 0.2.1 and its default prompts (one option order, lang matched to the clip):

Training

• Frozen: Qwen3-ASR-0.6B encoder and Qwen3-32B. Trained: the connector only.
• Content rounds (R1–R2): transcript distillation (token-level KL to the LLM's transcript-conditioned output, T = 2).
• Decision training: answer-token supervision, cross-entropy on the single option letter at the readout position. Transcript distillation alone cannot teach speaker cues, because its teacher never hears the voice.
• Data: public ASR/translation mixture: WenetSpeech, GigaSpeech, LibriSpeech, Common Voice, CoVoST (two packs of 0.5 M utterances).

Limitations

• Evaluated on read or acted speech and small test sets (100–800 items); not on streaming input.
• Paralinguistic training lowers accuracy on real spontaneous-style factual questions (see ZJU-ML above).
• Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
• The connector only works with the encoder and LLM listed above.

License

Apache-2.0 for the connector weights and code in this repository. The frozen base models keep their own licences (Qwen3-32B and Qwen3-ASR-0.6B: Apache-2.0). The training corpora have their own terms, and some of them (e.g. WenetSpeech, CoVoST 2) are licensed for non-commercial use only; check them before commercial use.

Citation

@inproceedings{jin2027duplexjev,
  title     = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author    = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  booktitle = {Submitted to IEEE ICASSP},
  year      = {2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.