MiniCPM-o 4.5 SimulS2ST: English → Chinese (r4 voice, 1,000 h)

A fine-tune of openbmb/MiniCPM-o-4_5 for simultaneous English-to-Chinese speech translation. The model listens to English speech in 1-second chunks. While the speaker is still talking, it writes the Chinese translation and speaks it in Chinese.

It was trained on about 1,000 hours of English speech from GigaSpeech XL, with Chinese text targets and Chinese speech targets in one voice (called "r4" below). The LoRA adapter (rank 32) is merged into the base weights, so this repo holds a full model: bf16, 17.7 GB. It handles speech only. The vision weights are not included.

Read this first

• The voice belongs to a real person. The model was trained to speak in the voice of one Mandarin reader from the FLEURS dataset (the FLEURS-R restored copy, CC BY 4.0). FLEURS does not name its readers, and we do not know whether this person agreed to have their voice cloned. Please do not use this model, or the speaker prompt in this repo, to imitate this person or anyone else, or to make speech that people may take for a real person speaking.
• Non-commercial use only. The weights are released under CC BY-NC 4.0, because the training audio (GigaSpeech) may be used only for non-commercial research and education. See Licence.
• The reported numbers need our streaming runtime. The scores below come from the Open-LiveTranslate inference runtime. It adds WAIT draining at the end of each phrase, a Talker that keeps its state across turns, and a sliding KV-cache window. This code is not public yet. The stock MiniCPM-o code loads these weights, but it runs the base model's own streaming loop, and we have not measured what that gives.

Results

Test data: the ACL 60/60 dev set, English → Chinese, all 5 talks (468 reference sentences). Each row is one run (seed 0) on one NVIDIA L40S. "Speed" means the English audio was played slower or faster with sox tempo.

• speech rows: the model's Chinese speech is transcribed with ElevenLabs scribe_v2 ASR and then scored.
• text rows: the model's own Chinese text from the same run, scored with no ASR.

How to read the table:

• CU and CA. CU (computation-unaware) places each output at the moment its English input had arrived. CA (computation-aware) also adds the time the model spent computing, so it is the closer match to a live system. BLEU, XCOMET-XL and TASER above come from the CU scoring. In the CA scoring, the speech rows get BLEU 48.49 (0.8x), 48.78 (1x), 46.99 (1.25x) and 40.65 (1.5x).
• BLEU. The output is split into sentences that match the reference sentences with SEGALE (the Speech-to-Speech-Latency toolkit). Both the reference and the output are NFKC-normalised before BLEU is computed. A reference sentence that gets no output is not dropped: it counts as 10 s of latency and an XCOMET score of 0. This happened for 0 of 468 sentences at 0.8x, 1x and 1.25x, and for 4 (speech) and 5 (text) of 468 at 1.5x.
• TASER (0-100) is an LLM judge (GPT-6-sol, reasoning effort high) that compares the output with the reference. It is not a human score.
• LongYAAL is the average delay per sentence, in milliseconds. Speech that goes on after the talk has ended is counted.
• End offset is how long after the English talk ends the Chinese output ends.
• Each number is from one run on a dev set. We have no test-set result for this model.

For context, here are two commercial live translation systems, measured with the same scorer at 1x. They report CA only, because their CU and CA audio is the same. They were fed 960 ms chunks instead of 1,000 ms.

Files

Using it

Requirements. We used transformers==4.51.0 and minicpmo-utils[tts] (the speech tokenizer and vocoder package of the base model). The speech decoder files (assets/token2wav, about 1.2 GB) are not in this repo. The model code downloads them from openbmb/MiniCPM-o-4_5 the first time it needs them, so the first run needs internet access. You can also copy that folder from the base repo.

Loading. We checked that the config, the fast tokenizer and the processor load, and that the weights match the model class (964 tensors, none missing or extra). We have not run generation through the stock code.

import torch
from transformers import AutoModel, AutoProcessor

repo = "owaski/MiniCPM-o-4_5-SimulS2ST-zh-r4-1000h"
model = AutoModel.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)

For the full-duplex streaming API, see the base model's README.

Inputs the model was trained with:

• System prompt, word for word: You are a professional simultaneous interpreter. You will be given chunks of English audio and you need to translate the audio into Chinese text.
• Audio fed in 1-second units.
• Speaker prompt: speaker_prompt/r4_fleursR24k.wav. The model was trained only on speech in this voice.

The evaluated setup (Open-LiveTranslate runtime, eval code commit 0749013):

• Thinker (text) decoding: greedy, at most 16 text tokens per 1 s unit.
• Talker (speech) sampling temperature: 0.2.
• Phrase ends: WAIT completion. After a phrase, the Talker sends WAIT and then quiet codes until new text arrives. At most 60 drain units, and up to 60 s of draining after the input ends.
• The Talker is continuous: it keeps its KV cache and vocoder state across turns.
• KV window: 24 units. Old units are dropped with nothing written into the prompt, and the kept units get new positions (context_no_previous_reindex).
• Warm-up: the model runs once on the first 10 s of the first input before the timed run starts.

Differences from the base model

• No vision. config.json has init_vision: false, and the 450 vision tensors (vpm, resampler) are not included. Do not switch vision on: it would get random weights.
• The base's banned-token list is missing. The base tokenizer_config.json lists 6,965 bad_token_ids that the stock duplex text decoder is never allowed to output. The ms-swift export dropped this list. Our evaluations loaded the tokenizer from this folder, so, as far as we can tell from the code, they ran without the list. We kept the files exactly as evaluated.
• Use the fast tokenizer (the default). tokenizer_config.json names a slow tokenizer whose file (tokenization_qwen2.py) is not included, so use_fast=False fails, and every load prints a warning about it. The fast tokenizer's vocabulary and tokenizer.json are identical to the base.
• Small config changes made by the export, which the evaluations ran with: tts_config.attn_implementation is eager (base: sdpa), audio_config.pad_token_id is 151643 (base: 50257), and audio_config.torch_dtype is bfloat16 (base: float32).

Training

Data.

• English audio: 243,329 training rows from a 1,000-hour subset of GigaSpeech XL (seed 42); 243,133 were used. The other 196 had no speech target.
• Chinese text targets:
  • Sentences re-cut and translated by Qwen3.8-27B-FP8 (greedy, temperature 0).
  • Rows kept only if the English ASR check passed (Parakeet TDT 0.6B v2, WER ≤ 0.1) and the translation passed MetricX-24 QE (score ≤ 3.0).
  • Words aligned with SimAlign (LaBSE).
  • The translation is released in whole phrases (at least 4 words, never across a sentence end) on a 1 s grid.
• Chinese speech targets:
  • Spoken by MOSS-TTS-v1.5 (OpenMOSS) in the r4 voice: 8 takes per request, keeping the take whose speaking rate is closest to the prompt's.
  • About 795 h of speech after cutting it into turns.
  • Speech codes from MiniCPM-o's own 25 Hz speech tokenizer; word timing from Qwen3-ForcedAligner-0.6B.

Method.

• Joint Thinker + Talker LoRA fine-tuning with ms-swift 3.9.1, starting from base revision 503e754207c94da6bb26850b4469f367c9ea3582.
• LoRA rank 32, alpha 64, dropout 0.05, on the q/k/v/o and MLP projections of every layer of the LLM (Qwen3-8B) and the Talker. audio_projection_layer and tts.projector_semantic were trained in full.
• Optimiser: AdamW, learning rate 1e-4 with a cosine schedule and no warm-up, weight decay 0.1.
• Length: 1 epoch = 15,196 steps, global batch 16 rows, at most 8,192 tokens per row, bf16, seed 42.
• Loss weights: text 1.0, speech 0.2. At most 16 text tokens per 1 s unit.
• WAIT supervision: after each phrase, the Talker learns to send WAIT and quiet codes until new text arrives.
• Hardware: 4 × NVIDIA L40S for 5.3 hours.

Provenance

Licence

• Weights, model card and training records: CC BY-NC 4.0. The weights are openbmb/MiniCPM-o-4_5 (Apache-2.0), changed by our fine-tuning.
• Code and tokenizer files from the base model: Apache-2.0. The text is in LICENSE-APACHE-2.0.
• Training audio: GigaSpeech. Its terms of access say: "Researcher shall use the Database only for non-commercial research and educational purposes." No GigaSpeech audio is in this repo.
• Speaker prompt: FLEURS-R, CC BY 4.0. The source and the changes are in speaker_prompt/LICENSE.
• Models used to make the training targets: Qwen3.8-27B-FP8, MOSS-TTS-v1.5 and Qwen3-ForcedAligner-0.6B (each Apache-2.0); MetricX-24, Parakeet TDT 0.6B v2 and LaBSE (see their own licences).

The full statement is in LICENSE. This is not legal advice.

Citations

If you use this model, please also cite the work it is built on:

• MiniCPM-o / MiniCPM-V: Yao et al., "MiniCPM-V: A GPT-4V Level MLLM on Your Phone", arXiv:2408.01800, 2024.
• GigaSpeech: Chen et al., "GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio", Interspeech 2021.
• FLEURS: Conneau et al., "FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech", SLT 2022.
• FLEURS-R: Ma et al., "FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks", Interspeech 2024.
• ACL 60/60: Salesky et al., "Evaluating Multilingual Speech Translation Under Realistic Conditions with Resegmentation and Terminology", IWSLT 2023.