Ke-Omni-R: Achieving Advanced Audio Reasoning with a Concise 50-Words Think Process

If you wish to train or perform inference with the model, please visit the GitHub repository: https://github.com/shuaijiang/Ke-Omni-R/. If you find this model helpful, please like this model and star our GitHub.

Ke-Omni-R is an advanced audio reasoning model built upon Qwen2.5-Omni-3B. With only 10k post-training samples, Ke-Omni-R has achieved state-of-the-art performance on the MMAU Test-mini and Test benchmarks. Key insights from its development include:

• GRPO Algorithm: The GRPO algorithm significantly enhances the performance of the already strong base model (Qwen2.5-Omni-7B), demonstrating superior generalization even in unseen speech domains.
• Think Process: Incorporating a concise think process (less than 50 words) plays a crucial role in improving reasoning capabilities.
• KL Divergence: Slight improvements were observed during GRPO training by leveraging KL divergence.
• Domain Ratio vs. Data Volume: Domain diversity outweighs data volume. We utilized only 10k samples, with 5k randomly selected from AVQA and another 5k from MusicBench.

Performance: Accuracies (%)↑ on MMAU Test-mini and Test benchmark

Performance: CER/WER (%)↓ on ASR benchmark

Note:

• * The data are sourced from the MMAU leaderboard.

• [1] Xie, Zhifei, et al. "Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models." arXiv preprint arXiv:2503.02318.

• [2] Ma, Ziyang, et al. "Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model." arXiv preprint arXiv:2501.07246.

• [3] Li, Gang, et al. "Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering." arXiv preprint arXiv:2503.11197

• [4] Xu, Jin, et al. "Qwen2.5-Omni Technical Report." arXiv preprint arXiv:2503.20215

Usage

from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info


# You can directly insert a local file path, a URL, or a base64-encoded audio into the position where you want in the text.
messages = [
  # Audio
    ## Local audio path
    [{"role": "system", "content":[{"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}]},
     {"role": "user", "content": [{"type": "audio", "audio": "/path_to_avqa_wavs/-IBtBeR6B00_000000.wav"}, {"type": "text", "text": "Please describe this audio."}]}],
    [{"role": "user", "content": [{"type": "audio", "audio": "/path_to_avqa_wavs/-IBtBeR6B00_000000.wav"}, {"type": "text", "text": "What is the main source of sound in the audio? ['aircraft', 'Car', 'Tank', 'Missile'] Output the thinking process (less than 50 words) in <think> </think> and final answer in <answer> </answer>."}]}],
    [{"role": "user", "content": [{"type": "audio", "audio": "/path_to_avqa_wavs/-IBXTktoom8_000030.wav"}, {"type": "text", "text": "What animal is the main source of sound in the video? ['dog', 'wasp', 'honeybee', 'dragonfly'] Output the thinking process (less than 50 words) in <think> </think> and final answer in <answer> </answer>."}]}],
]

model = Qwen2_5OmniForConditionalGeneration.from_pretrained('KE-Team/Ke-Omni-R-3B')
processor = Qwen2_5OmniProcessor.from_pretrained(model_path)

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(text)
audios, images, videos = process_mm_info(messages, use_audio_in_video=False)
inputs = processor(text=text, images=images, videos=videos, audio=audios, padding=True, return_tensors="pt")

generation = model.generate(**inputs, thinker_temperature=0, thinker_do_sample=False)
generated_ids = generation[:, inputs.input_ids.size(1):]
completions = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(completions)

the output should be

["Well, it sounds like there's a car accelerating. You can hear the engine revving up, and there's a bit of a thump or thud sound too. It might be the car hitting something or just a part of the acceleration process. It gives off a sense of speed and power. What do you think about it? Do you have any other audio samples you want to talk about?", '<think>The audio features a vehicle accelerating and revving, which is characteristic of a car. The sound is consistent with a car engine, not an aircraft, tank, or missile.</think>\n<answer>Car</answer>', "<think>The main source of sound is a buzzing insect, which is consistent with the size and sound of a honeybee. The other options don't match the sound or context.</think>\n<answer>honeybee</answer>"]

Acknowledgements

We express our gratitude to the following projects and teams for their contributions:

• R1-AQA: Referenced the GRPO-based training implementation from R1-AQA.
• Qwen Team: Special thanks to the Qwen2.5-Omni model for providing a robust foundation.
• Datasets:
  • AVAQ
  • MusicBench
  • MMAU

Citation

@misc{zhao2025keomnir,
  author = {Zhao, Shuaijiang and Guo, Tingwei and Wen, Cheng and Xiang, Bajian and Zou, Wei},
  title = {Ke-Omni-R: Achieving Advanced Audio Reasoning with a Concise 50-Words Think Process},
  year = {2025},
  publisher = {GitHub},
  journal = {GitHub Repository},
  howpublished = {\url{https://github.com/shuaijiang/Ke-Omni-R}},
}