DuplexJev-32B-Vision (research preview)

📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Examples: project page

DuplexJev hears what is said and how it is said. This version also sees. It takes a speech clip and a picture together and answers questions that need both. For example:

• Is the word in red the word the speaker stresses?
• Does the face show the same emotion as the voice?
• Does the caption describe the sound you hear?

As in every DuplexJev model, each answer is read as one constrained token (option orders averaged), so nothing is decoded. It can also reply in free text about the speech or the picture.

Results

Held-out test sets. A picture alone is 50% on every Yes/No task, so every point above 50 comes from hearing.

On the real photos, by the voice's emotion: happy 81.7, angry 59.2, sad 52.5. On the same frames, asked only "what emotion does the face show?" (3-way), the model gets 61.7% (chance 33%).

Use

vLLM (recommended)

pip install "vllm[audio]>=0.29" "duplexjev-vllm>=0.3"
vllm serve adventists-ai/DuplexJev-32B-Vision --max-model-len 8192 \
  --limit-mm-per-prompt '{"image": 1, "audio": 1, "video": 0}'

Standard OpenAI chat requests. Put <|audio|> in the text where the speech goes and an image_url part where the picture goes; read the answer as one constrained token:

req = {
    "model": "adventists-ai/DuplexJev-32B-Vision", "max_tokens": 1, "temperature": 0,
    "logprobs": True, "top_logprobs": 20, "allowed_token_ids": [32, 33],          # "A", "B"
    "chat_template_kwargs": {"enable_thinking": False},
    "messages": [{"role": "user", "content": [
        {"type": "text", "text": "The user said: <|audio|>\n\nPicture: "},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
        {"type": "text", "text": "\n\nDoes the face in the picture show the same emotion as the speaker's voice?"
                                 "\n\nOptions:\nA. Yes\nB. No\n\nAnswer with only the letter of the correct option."},
        {"type": "input_audio", "input_audio": {"data": "<base64 wav>", "format": "wav"}}]}],
}

Keep only the option letters from top_logprobs and renormalise; average over option orders for the scores below. A full client is in examples/vllm_vision_client.py. On the real-photo test, vLLM gives 64.7% (Transformers 64.4%; same decision on 353 of 360 pairs). One request with a picture and a 3 s clip takes about 0.05 s on one H200 (sequential; vLLM batches concurrent requests).

Transformers

pip install "duplexjev[vision]>=0.5"

from duplexjev.vision import VisionDecider

vd = VisionDecider.from_pretrained("adventists-ai/DuplexJev-32B-Vision")   # one 80 GB GPU

vd.decide(audio="clip.wav", image="face.jpg", questions=[
    {"id": "match", "text": "Does the face in the picture show the same emotion as the speaker's voice?",
     "options": ["Yes", "No"]}])
# {'match': {'answer': 'Yes', 'confidence': 0.99, 'probs': {'Yes': 0.99, 'No': 0.01}}}

vd.chat(image="photo.jpg", text="What is written in the picture?")

audio and image are both optional. Questions take the same form as in duplexjev.Decider (id, text, options).

The prompt the model was trained with (built for you by VisionDecider):

The user said: <|audio|>

Picture: <|vision_start|><|image_pad|>…<|vision_end|>

Does the face in the picture show the same emotion as the speaker's voice?

Options:
A. Yes
B. No

Answer with only the letter of the correct option.

Limitations

• The audio-visual LoRA was trained on synthetic pictures (rendered sentences, drawn faces, captions). On real photos it works best for happy faces and voices; angry and sad matches are often answered "No".
• Questions were trained in English. Speech can be Chinese or English.
• One image per request. Video is not supported.
• Free-text replies that combine the speech and the picture are less accurate than the multiple-choice readout; on real photos they sometimes misjudge the speaker's gender or age. Use multiple-choice questions for scored judgments.
• The speech-only decisions are those of DuplexJev-32B-Para v1.0. For speech alone, use DuplexJev-32B-Para (v1.1 is stronger).
• Emotion labels come from acted speech. Do not use the outputs to make decisions about individuals.

License

CC BY-NC 4.0, non-commercial use only (same training data as DuplexJev-32B-Para, see its card). Audio-visual items:

• synthetic pictures drawn for this model;
• audio from Stress-17K (CC BY-NC), VocalSound (CC BY-SA 4.0), AISHELL-1 (Apache-2.0) and LibriSpeech (CC BY 4.0).

Real-photo test frames and audio come from CREMA-D (ODbL / DbCL). Base models keep their licences: Qwen3-VL-32B-Instruct (including the vision encoder shipped here) and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud).

Citation

@misc{jin2026duplexjev,
  title         = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author        = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  year          = {2026},
  eprint        = {2610.02638},
  archivePrefix = {arXiv},
  note          = {Submitted to IEEE ICASSP 2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.