DuplexJev-B-Para-Whisper-small-Falcon-H1-3B
A small DuplexJev connector: the Whisper-small encoder (frozen) feeds a frozen Falcon-H1-3B, and every typed question is read as a single-token distribution over its options, with zero decode steps and many questions per forward pass. Trained with the same recipe as the Qwen3-32B connectors released with the paper.
| Audio encoder (frozen) | openai/whisper-small |
| LLM (frozen) | tiiuae/Falcon-H1-3B-Instruct |
| Connector | last encoder layer → frame stacking + SwiGLU projector (B), 30.4 M trainable parameters |
| Audio tokens | 50 Hz encoder frames, 8 stacked → 6.25 audio tokens/s |
| Total parameters | 3268 M (encoder + LLM + connector) |
| This variant | R2 connector + 4k steps of the four-task mixture (transcription, continuation, gender, emotion) under the mixed objective: transcript distillation for content samples, answer-token cross-entropy for decision samples (the paper's MIX-KD recipe). Hears speaker gender and four-way emotion. |
Quick start
pip install "duplexjev[speech]>=0.2.2"
from duplexjev import Decider, Question
d = Decider.from_pretrained("adventists-ai/DuplexJev-B-Para-Whisper-small-Falcon-H1-3B")
d.decide("utterance_en.wav", [
Question("gender", "What is the perceived gender of the speaker?", ["female", "male"], lang="en"),
Question("emotion", "What is the speaker's emotional state?", ["neutral", "happy", "angry", "sad"], lang="en"),
], lang="en")
d.decide("utterance_zh.wav", [ # Chinese speech: Chinese question and options
Question("gender", "说话人的性别是？", ["男性", "女性"], lang="zh"),
Question("emotion", "说话人当时的情绪状态是？", ["中性", "高兴", "生气", "伤心"], lang="zh"),
], lang="zh")
Falcon-H1 is a hybrid attention + Mamba model. Recurrent layers carry state along the sequence, so duplexjev (≥ 0.2.2) turns off prefix packing for it automatically (mode="batch": one row per question). For GPU speed install the Mamba kernels: pip install mamba-ssm causal-conv1d (they must be built for your torch/CUDA); without them transformers falls back to a slow reference path.
Results
| Evaluation (%) | paper protocol | duplexjev 0.2.2 defaults |
|---|---|---|
| qa100 (spoken QA) | 50 | 49 |
| qa100, Falcon-H1-3B reading the transcript | 81 | – |
| ZJU-ML (spoken QA, real + TTS) | 45 | – |
| Easy-Turn (4-way turn state, 800 clips) | 42.1 | – |
| Gender, 800 real utterances | 90.4 | 91.8 |
| Emotion, 4-way, 800 utterances | 79.5 | 77.6 |
Leaderboard scores (see the GitHub README): main language (mean of qa100, ZJU-ML, Easy-Turn) 45.7, paralinguistics (mean of gender, emotion) 85.0.
Paper protocol: the evaluation prompts of the paper (single option order, one H200). duplexjev defaults: the package's
own prompts with lang matched to the clip. Spoken QA with a small LLM is bounded by the LLM itself (see the
transcript-reading row); use these connectors for short decisions and speaker cues rather than open knowledge questions.
Latency of one decision event (10 questions, one 4.5 s clip, duplexjev 0.2.2, plain PyTorch): 169.7 s on 8 CPU threads (Xeon Platinum 8558, fp32, not quantized); about 104 ms on one H200 (bf16, measured while the GPU was shared with a training job). Falcon-H1 is not CPU-friendly: without Mamba CPU kernels transformers uses a slow reference path, and each question needs its own row.
Training
Frozen encoder and LLM; only the connector is trained: R1 and R2 transcript distillation on two disjoint 0.5 M-utterance packs of the Ultravox v0.6 mixture (32k steps each, global batch 16), then 4k steps of MIX-KD on content + gender (AISHELL-1, LibriSpeech) + emotion (ESD, CREMA-D). Code and scripts: https://github.com/adventists-ai/duplexjev.
Limitations
- Evaluated on read or acted speech and small test sets (100–800 items); not on streaming input.
- Small LLMs answer knowledge questions poorly even from the transcript.
- Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
- The connector only works with the encoder and LLM listed above.
License
CC BY-NC 4.0, research use only for the connector weights: trained on the ESD emotional speech corpus, which is licensed for research only, and on CREMA-D. Do not use it for commercial purposes. The frozen models keep their own licences: Falcon-H1-3B (TII Falcon License 2.0) and the encoder (Apache-2.0).
Citation
@inproceedings{jin2027duplexjev,
title = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
author = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
booktitle = {Submitted to IEEE ICASSP},
year = {2027}
}
Built by Adventists.ai. Claude (Anthropic) assisted with code.