PersonaPlex: Voice and role control for full duplex conversational speech models

➡️ Code: nvidia/personaplex ➡️ Demo: PersonaPlex Project Page ➡️ Paper: PersonaPlex Preprint

Description:

Personaplex is a real-time speech-to-speech conversational model that jointly performs streaming speech understanding and speech generation. The model operates on continuous audio encoded with a neural codec and predicts both text tokens and audio tokens autoregressively to produce its spoken responses. Incoming user audio is incrementally encoded and fed to the model while Personaplex simultaneously generates its own outgoing speech, enabling natural conversational dynamics such as interruptions, barge-ins, overlaps, and rapid turn-taking. Personaplex runs in a dual-stream configuration in which listening and speaking occur concurrently. This design allows the model to update its internal state based on the user’s ongoing speech while still producing fluent output audio, supporting highly interactive conversations. Before the conversation begins, Personaplex is conditioned on two prompts: a voice prompt and a text prompt. The voice prompt consists of a sequence of audio tokens that establish the target vocal characteristics and speaking style. The text prompt specifies persona attributes such as role, background, and scenario context. Together, these prompts define the model's conversational identity and guide its linguistic and acoustic behavior throughout the interaction.

This model is ready for commercial use.

Explore more from NVIDIA:

For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at developer.nvidia.com. Join the community to access tools, support, and resources to accelerate your development with NVIDIA's NeMo, Riva, NIM, and foundation models.

What is Nemotron? NVIDIA Developer Nemotron NVIDIA Riva Speech NeMo Documentation

License/Terms of Use:

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License Agreement. ADDITIONAL INFORMATION: CC-BY-4.0.

Use Case:

Wherever NVIDIA’s speech-to-speech conversational models are used, PersonaPlex can generate English speech response for English speech input.

Deployment Geography:

Global

Release Date:

Hugging Face [01/15/2026] via [https://huggingface.co/nvidia/personaplex-7b-v1] Github [01/15/2026] via [https://github.com/NVIDIA/personaplex]

Model Architecture:

Architecture Type: Transformer

Network Architecture: Moshi

Moshi uses:

• Mimi Speech Encoder (ConvNet, Transformer)
• Moshi Temporal Transformer + Depth Transformer
• Mimi Speech Decoder (Transformer, ConvNet)

** This model was developed based on Moshi (Moshiko weights) ** Number of model parameters: 7B

Input(s):

Input Type(s): Text (prompt), Audio (user speech) Input Format: String, WAV/WebAudio Input Parameters: One-Dimensional (1D) Other Properties Related to Input: 24kHz sample rate for audio.

Output(s)

Output Type(s): Text (agent text), Audio (agent speech) Output Format: String, WAV/WebAudio Output Parameters: One-Dimensional (1D) Other Properties Related to Output: 24kHz sample rate for audio.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine: PyTorch

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere (A100)
• NVIDIA Hopper (H100)

Preferred/Supported Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

• v1.0

Training, Testing, and Evaluation Datasets:

Training Dataset:

Link: Fisher English: Part1, Part2 Data Modality: Audio (speech) Audio Training Data Size: Less than 10,000 Hours Data Collection Method by dataset: Human Labeling Method by dataset: Automated Properties: 7303 conversations (upto 10 minutes each).

Testing/Evaluation Dataset:

Link: FullDuplexBench Data Collection Method by dataset: Hybrid: Human, Synthetic, Automated. Labeling Method by dataset: Automated. Properties: The FullDuplexBench public benchmark aggregates various synthetic and real datasets. Additionally speaker similarity (SSIM) between voice prompts and model outputs on the User Interruption portion of the FullDuplexBench benchmark were measured using WavLM-TDNN embedding cosine similarity.

FullDuplexBench Benchmark Scores:

Comparison With Other Conversational AI Systems: PersonaPlex outperforms other open-source and commercial systems on conversational dynamics, response and interruption latency, and task adherence in both question-answering assistant and customer service roles.

Inference:

Acceleration Engine: PyTorch Test Hardware: NVIDIA A100 80 GB

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Citation

If you use PersonaPlex in your research, please cite our paper:

@misc{roy2026personaplexvoicerolecontrol,
      title={PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models}, 
      author={Rajarshi Roy and Jonathan Raiman and Sang-gil Lee and Teodor-Dumitru Ene and Robert Kirby and Sungwon Kim and Jaehyeon Kim and Bryan Catanzaro},
      year={2026},
      eprint={2602.06053},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.06053}, 
}

References(s):

1. Moshi and Mimi
2. FullDuplexBench