MERaLiON-Omni-GRPO-10B

Resource reproduction following the paper:

"Unlocking Cognitive Capabilities and Analyzing the Perception-Logic Trade-off"

Model

MERaLiON-Omni-GRPO-10B is the culturally-aligned variant, fine-tuned with dual-reward Group Reward Optimization for enhanced cultural density and multimodal coherence. Accepts image, video, and audio inputs. Multilingual and reasoning-enhanced variants of the model will be released in upcoming updates.

Architecture: Same base as MERaLiON-Omni-MM-10B, fine-tuned with LoRA (r=8, alpha=32) via dual-reward GRPO (Cultural Density λ=0.2 + Multimodal Coherence).

This model is especially suited for tasks requiring cultural nuance in Southeast Asian contexts.

Important Notice

• Alpha Release. This is MERaLiON2-Omni (Alpha). The model was trained with SEA images + SEA audios + general videos, so its video understanding for Southeast Asia is zero-shot and research-oriented.
• Not Production-Ready. This model has not undergone safety alignment or red-teaming post-training. It may produce hallucinated, biased, or otherwise unreliable outputs. It is not suitable for real-world deployment.
• Research Use Only. This release is intended for academic research and evaluation purposes only. Commercial use is not permitted. See LICENSE for full terms.

License

This model is released under a Research-Only License. Commercial use is prohibited without explicit authorization.

Demo

Usage

Installation and inference interface is identical to MERaLiON-Omni-MM-10B:

Requirements

Tested with the following environment:

pip install transformers torch pillow torchaudio flash-attn triton

Load model

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
from mm_utils_mm import process_mm_info  # bundled in this repository

model_path = "zzlynxSG/MERaLiON-Omni-GRPO-10B"

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
processor.tokenizer.padding_side = "left"
if processor.tokenizer.pad_token is None:
    processor.tokenizer.pad_token = processor.tokenizer.eos_token

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_path, trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    use_safetensors=True,
    device_map="auto"
)

MIN_PIX = 4 * 28 * 28
MAX_PIX = 8192 * 28 * 28
MAX_PIX_VIDEO = 8600 * 28 * 28

Image

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=[[{"role": "user", "content": "Follow the text instruction based on the following image: <ImageHere> \n Describe the cultural elements in this image."}]],
    tokenize=False, add_generation_prompt=True
)[0]

mm_input = [{"role": "user", "content": [
    {"type": "image", "image": "your_image.jpg", "min_pixels": MIN_PIX, "max_pixels": MAX_PIX}
]}]
audios, images, videos = process_mm_info(mm_input)
inputs = processor(text=[chat_prompt], audios=audios, images=images, videos=videos).to(model.device).to(model.dtype)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(processor.decode(outputs[0], skip_special_tokens=True))

Audio

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=[[{"role": "user", "content": "Follow the text instruction based on the following audio: <SpeechHere> \n Transcribe this audio."}]],
    tokenize=False, add_generation_prompt=True
)[0]

mm_input = [{"role": "user", "content": [
    {"type": "audio", "audio": "your_audio.mp3"}
]}]
audios, images, videos = process_mm_info(mm_input)
inputs = processor(text=[chat_prompt], audios=audios, images=images, videos=videos).to(model.device).to(model.dtype)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(processor.decode(outputs[0], skip_special_tokens=True))

Video + audio

chat_prompt = processor.tokenizer.apply_chat_template(
    conversation=[[{"role": "user", "content": "Follow the text instruction based on the following video: <VideoHere> \n Summarize this video."}]],
    tokenize=False, add_generation_prompt=True
)[0]

mm_input = [{"role": "user", "content": [
    {"type": "video", "video": "your_video.mp4", "fps": 2, "min_frames": 32, "max_frames": 64,
     "min_pixels": MIN_PIX, "total_pixels": MAX_PIX_VIDEO}
]}]
audios, images, videos = process_mm_info(mm_input, use_audio_in_video=True)
inputs = processor(text=[chat_prompt], audios=audios, images=images, videos=videos).to(model.device).to(model.dtype)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(processor.decode(outputs[0], skip_special_tokens=True))