DuplexJev-Gemma-31B
📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev
Built on Gemma-4-31B-it + the Qwen3-ASR-0.6B audio encoder. The non-Qwen member of the DuplexJev family: strongest English reasoning of the LLMs we tested, Apache-2.0 base.
DuplexJev reads typed, closed-set decisions from speech (has the user finished? who is speaking, in what mood?) as a single-token distribution of the LLM: zero decode steps, many questions about the same clip in parallel. This repository is the complete model (encoder + trained connector + LLM, bf16, about 58 GB) and runs on vLLM with a small plugin.
| Audio encoder (frozen) | Qwen3-ASR-0.6B encoder, 12.5 frames/s |
| Connector (trained) | 2 frames stacked → 6.25 audio tokens/s → SwiGLU MLP → LLM width; audio embeddings ×3 and wrapped in Gemma's native <|audio> … <audio|> markers |
| LLM | Gemma-4-31B-it text model. Unlike DuplexJev-32B / 4B, the LLM is not frozen: a LoRA (r 16, α 32, q/k/v/o) was trained in the paralinguistic stage and merged into the weights |
| Training | content alignment (transcript distillation, LLM frozen) → LoRA content warm-up → gender + emotion with LoRA under the mixed objective (CE on decisions, KL on content) |
| Inputs | audio + text. Gemma's vision tower is not included. |
Serve with vLLM
pip install "vllm[audio]>=0.29" "duplexjev-vllm>=0.2.0"
vllm serve adventists-ai/DuplexJev-Gemma-31B --max-model-len 4096
One 80 GB GPU is enough (or --tensor-parallel-size 2). duplexjev-vllm ≥ 0.2.0 registers
DuplexJevGemmaForConditionalGeneration (Qwen3-ASR encoder + connector + ×3 scale + audio markers; the LLM runs on
vLLM's own Gemma kernels). The placeholder in prompts stays <|audio|>; the plugin expands it.
Each question is one chat request with max_tokens=1: list the options under letters and read the log-probabilities
of the letters. A ready-made client is
examples/vllm_client.py:
from vllm_client import DuplexJevClient
dj = DuplexJevClient("http://localhost:8000/v1")
dj.decide(open("call.wav", "rb").read(), {
"gender": ("What is the perceived gender of the speaker?", ["female", "male"]),
"emotion": ("What is the speaker's emotional state?", ["neutral", "happy", "angry", "sad"]),
})
The prompt format the connector was trained on:
The user said: <|audio|>
Question: What is the perceived gender of the speaker?
Options:
A. female
B. male
Answer with only the letter of the correct option.
For Chinese speech ask in Chinese: 用户说：<|audio|>, 问题：…, 选项：, 请只回答正确选项的字母。
Results
Served by vLLM 0.29 with duplexjev-vllm 0.2.0, questions in the training wording above (one H200, bf16; % correct):
| model | qa100 | ZJU-ML | Easy-Turn | gender | emotion |
|---|---|---|---|---|---|
| DuplexJev-Gemma-31B (this repo) | 97 | 87 | 68.0* | 94.4 | 90.6 |
| DuplexJev-32B | 94 | 88 | 79.4 | 89.8 | 89.9 |
| DuplexJev-4B | 77 | 49 | 60.0 | 87.9 | 90.9 |
* No turn-taking data was used for this model; Easy-Turn is zero-shot here (DuplexJev-32B saw the Easy-Turn training split).
Wording sensitivity. With a question wording that differs from training (our paper-protocol evaluation script), this model scores qa100 96, ZJU-ML 84, but gender 78.5 and emotion 60.9. The DuplexJev-32B connector (frozen LLM) is much less sensitive to wording. Use the prompt format above.
qa100: 100 bilingual spoken multiple-choice questions (adventists-ai/qa100).
ZJU-ML: main-language part of the ZJU audio benchmark v2.0.0. Easy-Turn: 800-item four-way turn-state test set.
Gender: 800 real utterances (AISHELL-1, LibriSpeech). Emotion: 800 utterances (ESD, CREMA-D).
Limitations
- Evaluated on short read or acted speech; not on streaming input.
- Paralinguistic accuracy depends on prompt wording (see above).
- No turn-taking training yet; prefer DuplexJev-32B for end-of-turn decisions.
- Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
License
CC BY-NC 4.0, research use only. The connector and LoRA were trained on the ESD emotional speech corpus, which is licensed for research purposes only, and on CREMA-D (ODbL). The base models keep their own licences: Gemma-4-31B-it (Apache-2.0, Google) and Qwen3-ASR-0.6B (Apache-2.0, Alibaba Cloud). The LLM weights in this repository are Gemma-4-31B-it's text-model weights with a LoRA merged in; the vision tower is not included.
Citation
@misc{jin2026duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
year = {2026},
eprint = {2610.02638},
archivePrefix = {arXiv},
note = {Submitted to IEEE ICASSP 2027}
}
Built by Adventists.ai. Claude (Anthropic) assisted with code.