JoyFox-OmniFlow-9B

JoyFox-OmniFlow-9B is a general-purpose, full-duplex multimodal model built on OpenBMB MiniCPM-o 4.5.

The model is designed for:

• English, Chinese, and multilingual conversations;
• instruction following and complex question answering;
• mathematical reasoning and code generation;
• knowledge-intensive and long-form responses;
• role-play, character-driven dialogue, and natural conversation;
• agent workflows and tool-calling formats;
• real-time vision, audio, text, and speech interaction.

JoyFox-OmniFlow-9B is distributed as a complete BF16 Safetensors model that can be loaded directly with Hugging Face Transformers. The release includes the language model, vision encoder, audio encoder, multimodal projection layers, TTS module, voice-cloning support, token-to-waveform resources, and full-duplex streaming components.

Highlights

• Complete full-duplex model: supports concurrent visual/audio input and text/speech output.
• Integrated TTS: includes speech generation weights, token2wav resources, and reference audio assets.
• Broad language capabilities: covers general instruction following, math, code, multilingual dialogue, role-play, and tool use.
• Omni-modal interaction: combines language, vision, audio, multimodal projection, and streaming session modules.
• Voice cloning and character voices: supports reference-audio-driven speech generation.
• Ready for GPU inference: fully validated on an NVIDIA RTX PRO 6000 Blackwell GPU.
• Standard Transformers package: no additional model-weight package is required at load time.

Capability Overview

Model Package

Integrity Validation

The released model passed the following checks:

• all 1,414 model weight keys are readable;
• all 194 TTS weight tensors are readable;
• the token2wav runtime assets are present;
• init_audio=true;
• init_vision=true;
• init_tts=true;
• stream_input=true;
• zero missing keys during a complete Transformers load;
• zero unexpected keys during a complete Transformers load;
• zero mismatched keys during a complete Transformers load.

GPU Load Validation

The complete BF16 model was successfully loaded on an NVIDIA RTX PRO 6000 Blackwell Server Edition:

• llm: CUDA
• vpm: CUDA
• apm: CUDA
• tts: CUDA
• audio_projection_layer: CUDA
• resampler: CUDA
• model-weight VRAM usage: approximately 17.56 GiB

The VRAM figure above primarily reflects model weights and small framework overhead. Full-duplex inference requires additional memory for the KV cache, video frames, audio buffers, and token2wav.

Installation

For text, image, or audio-understanding inference:

pip install "transformers==4.51.0" accelerate \
  "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" \
  "minicpmo-utils>=1.0.5"

For TTS or full-duplex streaming inference:

pip install "transformers==4.51.0" accelerate \
  "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" \
  "minicpmo-utils[all]>=1.0.5"

FFmpeg is also required for full-duplex video processing:

sudo apt update
sudo apt install -y ffmpeg

The model structure was also validated locally with transformers==4.57.0 and torch==2.11.0+cu130. For production full-duplex use, the MiniCPM-o 4.5 officially recommended dependency versions are preferred.

GPU Quick Start

import torch
from transformers import AutoModel, AutoProcessor

model_id = "joyfox/JoyFox-OmniFlow-9B"

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=True,
    init_tts=True,
)
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval().cuda()

Enabling TTS and Full-Duplex Mode

After installing minicpmo-utils[all]:

# Initialize token2wav for speech-token-to-waveform decoding.
model.init_tts()

# Enter full-duplex mode.
duplex_model = model.as_duplex()

# Return to standard half-duplex mode when needed.
model = duplex_model.as_simplex(reset_session=True)

Real-time video/audio chunking, streaming_prefill, streaming_generate, and output-video generation follow the standard MiniCPM-o 4.5 full-duplex API.

Speech Output Example

import librosa
import torch
from huggingface_hub import hf_hub_download

ref_audio_path = hf_hub_download(
    repo_id=model_id,
    filename="assets/HT_ref_audio.wav",
)
ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)

model.init_tts()
system_message = model.get_sys_prompt(
    ref_audio=ref_audio,
    mode="omni",
    language="en",
)

messages = [
    system_message,
    {
        "role": "user",
        "content": ["Introduce yourself in a natural and concise way."],
    },
]

with torch.inference_mode():
    answer = model.chat(
        msgs=messages,
        do_sample=True,
        temperature=0.7,
        max_new_tokens=1024,
        use_tts_template=True,
        enable_thinking=False,
        omni_mode=True,
        generate_audio=True,
        output_audio_path="output.wav",
    )

print(answer)

Intended Use

JoyFox-OmniFlow-9B is suitable for:

• general English, Chinese, and multilingual assistants;
• mathematical, coding, knowledge, and complex instruction tasks;
• role-play and character-driven dialogue;
• agent and tool-calling applications;
• multimodal assistants with visual and audio input;
• half-duplex conversations with speech output;
• real-time full-duplex video and audio interaction;
• reference-audio-based voice cloning and character speech.

Limitations

• This release does not report independent external benchmark results.
• Response style depends on the system prompt, chat template, and decoding configuration.
• Production tool-calling systems should validate schemas, arguments, and execution permissions.
• The model may produce inaccurate, biased, or inappropriate content; high-risk use cases require human review.
• Full-duplex quality depends on audio sampling, video chunking, system prompts, reference voices, and runtime implementation.
• Full-duplex TTS requires the additional runtime dependencies included with minicpmo-utils[all].

License and Acknowledgements

This model is built on MiniCPM-o 4.5 and is subject to the applicable upstream license requirements. The model card declares Apache-2.0 metadata. Before deployment, redistribution, or commercial use, review the original MiniCPM-o 4.5 license and the terms applicable to your intended use.

We thank OpenBMB, LLaMA-Factory, and the open-source community for the model, tooling, and supporting resources used in this project.