X2-Turn-4B-0812

Real-time bilingual ASR with turn-taking prediction for voice assistants.

Online demo: x-square-robot/x2-turn-4b-demo — try the model in your browser (no local GPU required). Thanks to @multimodalart for creating the Space.

X2-Turn-4B-0812 listens to live speech and produces two synchronized outputs:

1. Streaming transcription in Chinese and English (including mixed speech)
2. Turn-state predictions every 80 ms — whether the user is silent, still speaking, finished, or offering a brief backchannel

Built on mistralai/Voxtral-Mini-4B-Realtime-2602, this checkpoint adds an independent turn-prediction head while preserving the original ASR backbone. It is designed for voice assistants that must decide when to wait, respond, ignore a backchannel, or allow interruption.

Model summary

Quick start

Install the inference wrapper from the X2-Turn code repository. No trust_remote_code is required.

git clone https://github.com/X-Square-Robot/X2-Turn.git
python -m pip install -e "./X2-Turn[transformers]"

import torch
from transformers import AutoProcessor
from voxtral_realtime.transformers import infer_asr_turn, load_mtp_checkpoint

model_id = "x-square-robot/X2-Turn-4B-0812"

processor = AutoProcessor.from_pretrained(model_id)
model = load_mtp_checkpoint(
    model_id,
    device="cuda",
    dtype=torch.bfloat16,
).eval()

result = infer_asr_turn(model, processor, "/path/to/input.wav")

print("ASR:", result.transcript)
for frame in result.turn_frames:
    print(frame.start_ms, frame.end_ms, frame.label, frame.confidence)

• result.transcript — recognized text
• result.turn_frames — turn label and confidence for each 80 ms frame

The loader also accepts a local checkpoint directory. For a full command-line example:

python examples/offline_inference.py \
  --model x-square-robot/X2-Turn-4B-0812 \
  --audio /path/to/input.wav \
  --output offline_frames.json

Turn labels

At inference time, the model emits one of six turn states per 80 ms frame. Five of these correspond to the turn states defined and trained in the paper; uncertain is an inference-only label used when the model is not confident that the turn has ended — it was not used during training.

Paper ↔ inference label mapping

Paper metrics such as ACCcomp and ACCincomp in Table 1 refer to <\|complete\|> and <\|incomplete\|>, which map to turn_end and speaking respectively at inference time.

These are predictions, not commands. Production systems should smooth across several frames and apply a policy rather than acting on a single frame.

Benchmark results

Evaluation results are reported in X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction (arXiv:2608.10878). This checkpoint is evaluated at streaming delay τ = 480 ms.

Table 1 — Turn state classification on EasyTurn (τ = 480 ms)

Table 2 — Effect of streaming delay τ on EasyTurn

For the full baseline comparison (including cascaded systems), ASR results (Table 3), and the dual-head architecture overview (Figure 1), see the paper PDF.

Use cases

Recommended for:

• Low-latency Mandarin, English, and mixed-language ASR
• Voice-assistant response timing and endpointing
• Distinguishing real requests from backchannels
• Barge-in detection and turn-taking experiments
• Controlled research and product evaluation with monitoring

Not recommended for:

• Safety-critical decisions without human oversight
• Speaker identity or emotion inference
• Legal transcription or covert surveillance as the sole basis for action

Realtime serving

The fastest way to try the model is the hosted Space: x-square-robot/x2-turn-4b-demo.

Production deployment uses the X2-Turn code repository with a pinned vLLM overlay. Stock vLLM does not emit the custom turn.delta events — follow the vLLM integration guide before serving.

For local browser-based visualization of ASR, turn frames, and ACCEPT/REJECT/HOLD/barge-in decisions, see the turn-demo component in the repository.

Architecture

This checkpoint extends Voxtral-Mini-4B-Realtime-2602 with:

• Shared backbone and ASR lm_head (stored under base_model.*)
• Independent turn head vad_lm_head.weight with full vocabulary

Turn labels map to reserved tokenizer IDs 35–40. The five trained paper labels (idle, noidle, incomplete→speaking, complete→turn_end, backchannel) occupy IDs 35–39; uncertain (ID 40) is reserved for inference-only low-confidence fallback and was not supervised during training.

The canonical model.safetensors uses a single-file layout. The voxtral_realtime.transformers loader creates the VoxtralMTP wrapper before loading both heads. Runtime metadata includes params.json, tekken.json, processor_config.json, and generation_config.json.

Limitations

• Accuracy may degrade under noise, reverberation, overlapping speakers, accents, dialects, code-switching, far-field microphones, or packet loss.
• Turn predictions can flicker or arrive early/late; temporal smoothing is recommended.
• ASR errors and turn errors interact — incomplete text does not always mean an incomplete turn.
• Performance may vary across demographic groups, speaking styles, languages, microphones, and environments.
• Real-time latency depends on hardware, serving configuration, and policy buffering.

Speech may contain personal, biometric, confidential, or copyrighted information. Obtain appropriate consent, minimize collection and retention, and avoid logging raw audio or transcripts by default.

Citation

If you use this model, please cite:

@article{fu2026x2turn,
  title   = {X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction},
  author  = {Kaiqi Fu and Rime Wen and Altman Lin and Shawn Qin and Roy Gan and Hao Wang and Qian Wang},
  journal = {arXiv preprint arXiv:2608.10878},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.10878}
}

Acknowledgments

Thanks to @multimodalart from Hugging Face for building the online demo.

License

Model weights and code are released under Apache-2.0. Use is also subject to the Mistral base model terms. See NOTICE for attribution details.