DuplexJev-B-SenseVoice-Small-Qwen3-4B
A small DuplexJev connector: the SenseVoice-Small encoder (frozen) feeds a frozen Qwen3-4B, and every typed question is read as a single-token distribution over its options, with zero decode steps and many questions per forward pass. It is the edge-sized counterpart of the Qwen3-32B connectors released with the paper, trained with the same recipe.
| Audio encoder (frozen) | adventists-ai/SenseVoice-Small-Encoder |
| LLM (frozen) | Qwen/Qwen3-4B |
| Connector | last encoder layer → frame stacking + SwiGLU projector (B), 11.5 M trainable parameters |
| Audio tokens | SenseVoice at 16.7 Hz (60 ms LFR frames), 3 frames stacked → 5.6 audio tokens/s |
| Total parameters | 4255 M (encoder + LLM + connector) |
| This variant | Content alignment only (R1–R2, transcript distillation). No paralinguistic training: gender and emotion are at chance. A starting point for your own decision fine-tuning. |
Quick start
pip install "duplexjev[speech]>=0.2.1" torchaudio # the SenseVoice encoder needs torchaudio
from duplexjev import Decider, Question
d = Decider.from_pretrained("adventists-ai/DuplexJev-B-SenseVoice-Small-Qwen3-4B") # fits on one small GPU or CPU
d.decide("utterance_en.wav", [
Question("turn", "Has the user finished speaking?", ["finished", "not finished"], lang="en"),
Question("intent", "What does the user want?", ["climate", "media", "navigation", "phone", "chit-chat"], lang="en"),
], lang="en")
d.decide("utterance_zh.wav", [ # Chinese speech: Chinese question and options
Question("turn", "用户说完了吗？", ["说完了", "还没说完"], lang="zh"),
Question("intent", "用户想做什么？", ["空调", "媒体", "导航", "电话", "闲聊"], lang="zh"),
], lang="zh")
Results
| Evaluation (%) | paper protocol | duplexjev 0.2.1 defaults |
|---|---|---|
| qa100 (spoken QA) | 71 | 74 |
| qa100, Qwen3-4B reading the transcript | 82 | – |
| ZJU-ML (spoken QA, real + TTS) | 57 | – |
| Gender, 800 real utterances | 53.4 | 52.9 |
| Emotion, 4-way, 800 utterances | 26.9 | 26.9 |
Paper protocol: the evaluation prompts of the paper (single option order, one H200). duplexjev defaults: the package's
own prompts with lang matched to the clip. Spoken QA with a small LLM is sensitive to the prompt wording, and it is
bounded by the LLM itself (see the transcript-reading row); use these connectors for short decisions and speaker cues
rather than open knowledge questions. For comparison, the Qwen3-32B connectors reach 90 (qa100), 89.9 (gender) and
90.0 (emotion); see adventists-ai.
Latency of one decision event (10 questions, one 4.5 s clip, one forward pass, duplexjev 0.2.1, plain PyTorch):
149 ms on one H200 (bf16); 6057 ms on 8 CPU threads (Xeon Platinum 8558, fp32, not quantized).
On-device NPUs and quantization have not been tested yet.
Training
Frozen encoder and LLM; only the connector is trained, with the paper's recipe: R1 and R2 transcript distillation on two disjoint 0.5 M-utterance packs of the Ultravox v0.6 mixture (32k steps each, global batch 16). Code and scripts: https://github.com/adventists-ai/duplexjev.
Limitations
- Evaluated on read or acted speech and small test sets (100–800 items); not on streaming input.
- Small LLMs answer knowledge questions poorly even from the transcript.
- Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
- The connector only works with the encoder and LLM listed above.
License
Apache-2.0 for the connector weights. The frozen base models keep their own licences. Some training corpora (e.g. WenetSpeech, CoVoST 2) are licensed for non-commercial use only; check them before commercial use.
The SenseVoice-Small encoder (Alibaba Group, FunAudioLLM) is redistributed under the FunASR Model Open Source License; keep its attribution and model name.
Citation
@inproceedings{jin2027duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
booktitle = {Submitted to IEEE ICASSP},
year = {2027}
}
Built by Adventists.ai. Claude (Anthropic) assisted with code.