TLive-Omni

📋 Overview

TLive-Omni is an omni-modal understanding model for e-commerce live-stream, mapping image, video, audio, and text into a unified text-output interface. Built on a Qwen3.5 backbone with a grafted AuT audio encoder, it supports up to 256K tokens of context, trained via a three-stage SFT recipe followed by Faithful-RFT reinforcement fine-tuning.

✨ Highlights

• Timestamped Per-vGrid layout — Audio and video tokens are organized into timestamped grid with explicit boundaries, keeping audio segments adjacent to their corresponding visual content for fine-grained temporal alignment over long streams.
• Three-stage SFT recipe — Progressive training from audio-language alignment to full multimodal SFT, developing live-commerce understanding from omni-modal perception to instruction-following responses.
• Faithful-RFT — A reinforcement fine-tuning stage for faithful and real-time live-stream demands, suppressing explicit reasoning traces and directly optimizing answer quality for live-commerce tasks.
• Rich atomic capabilities — A scenario-oriented taxonomy covering speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc, supported by a compact data production engine.
• Strong live-commerce performance with competitive generalization — 4B and 9B variants demonstrate strong results across live-commerce audio, image, and video tasks, together with excellent generalization on general benchmarks.

🏗️ Architecture

TLive-Omni is built on a Qwen3.5 backbone and extends it with a audio encoder through a lightweight MLP aligner, forming a unified text-output omni-modal understanding model. For video inputs with audio, each temporal grid is organized into a timestamped grid that interleaves video and audio token blocks, keeping audio segments adjacent to their corresponding visual content. The model supports up to 256K tokens of context at inference.

📊 Benchmark Results

We evaluate TLive-Omni-4B and TLive-Omni-9B on both live-commerce tasks and general benchmarks. Dash (-) denotes an unreported result or undisclosed parameter count. The Best results among the compared open-source models are marked in bold, while the second-best results are in underlined.

Live-Commerce Evaluation

General Benchmark: Image Understanding

General Benchmark: Video Understanding

General Benchmark: Omni Understanding

⚙️ Installation

This release targets Python 3.10 on Linux x86_64 with CUDA 12.8 and PyTorch 2.10.0.

conda create -n tlive python=3.10 -y
conda activate tlive
pip install -r https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/v1.0.0-rc1/environments/requirements.txt

The remote environments/requirements.txt includes custom wheels for the supported environment and model. If any wheel does not match your hardware, CUDA version, or Python version, replace it with a compatible build for your setup.

🚀 Quick Start

Transformers inference

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "TaoLiveAIGC/TLive-Omni-4B"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
).eval()


def generate(messages, *, use_audio_in_video=False, videos_kwargs=None, generation_kwargs=None):
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        enable_thinking=False,
        use_audio_in_video=use_audio_in_video,
        videos_kwargs=videos_kwargs or {},
    )
    prompt_length = inputs["input_ids"].shape[-1]
    inputs = inputs.to(model.device)

    generation_kwargs = generation_kwargs or {}
    with torch.inference_mode():
        generated_ids = model.generate(
            **inputs,
            do_sample=False,
            max_new_tokens=1024,
            **generation_kwargs,
        )

    answer_ids = generated_ids[:, prompt_length:]
    answer = processor.batch_decode(
        answer_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]
    return answer.strip()

Replace messages with one of the examples below for text, image, audio, or video inputs.

Text

messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "Briefly explain why multimodal context can improve an answer."}],
}]
print(generate(messages))

Image

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/image.jpg"},
        {"type": "text", "text": "Describe this image."},
    ],
}]
print(generate(messages))

Audio

messages = [{
    "role": "user",
    "content": [
        {"type": "audio", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/audio.mp3"},
        {"type": "text", "text": "Transcribe and summarize this audio."},
    ],
}]
print(generate(messages))

Video with audio

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/vocal_video.mp4"},
        {"type": "text", "text": "Describe the video, including relevant speech and sounds."},
    ],
}]
print(generate(messages, use_audio_in_video=True, videos_kwargs={"fps": 1.0}))

Video without audio

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "path": "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/silence_video.mp4"},
        {"type": "text", "text": "Describe the visual events in this video."},
    ],
}]
print(generate(messages, use_audio_in_video=False, videos_kwargs={"fps": 1.0}))

For temporal localization outputs, we recommend the MM:SS - MM:SS interval format, for example 01:23 - 01:35. For videos, set use_audio_in_video=True when the audio track should be used, and False for visual-only inference.

⚡ vLLM

Installation

First install the pre-built wheel (Python 3.10 + CUDA 12.8 + Linux x86_64), built and tested on NVIDIA H20 GPUs (Hopper, sm_90):

pip install https://github.com/TaoLiveAIGC/TLive-Omni/releases/download/v1.0.0-rc1/vllm-0.19.0+cu128-cp310-cp310-linux_x86_64.whl

If your GPU, driver, or CUDA setup is not compatible with this wheel, build vLLM from source using the customized code in the vllm/ directory of the GitHub release.

Inference

from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from vllm.model_executor.models.tlive_omni_processing import process_audio_info

model_id = "TaoLiveAIGC/TLive-Omni-4B"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)


def build_prompt(messages):
    return processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=False,
        enable_thinking=False,
    )


def generate(inputs, *, limit_mm_per_prompt=None, vllm_kwargs=None, sampling_kwargs=None):
    vllm_kwargs = vllm_kwargs or {}
    sampling_kwargs = sampling_kwargs or {}
    llm = LLM(
        model=model_id,
        trust_remote_code=True,
        dtype="bfloat16",
        max_model_len=32768,
        tensor_parallel_size=1,
        gpu_memory_utilization=0.9,
        max_num_seqs=4,
        max_num_batched_tokens=32768,
        seed=42,
        limit_mm_per_prompt=limit_mm_per_prompt,
        **vllm_kwargs,
    )
    outputs = llm.generate(
        inputs,
        sampling_params=SamplingParams(
            temperature=0.0,
            max_tokens=1024,
            **sampling_kwargs,
        ),
    )
    return outputs[0].outputs[0].text.strip()

Replace messages with one of the examples below for text, image, audio, or video inputs.

Text

messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "Briefly explain why multimodal context can improve an answer."}],
}]
inputs = {"prompt": build_prompt(messages)}
print(generate(inputs))

Image

image_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/image.jpg"
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "path": image_path},
        {"type": "text", "text": "Describe this image."},
    ],
}]
inputs = {
    "prompt": build_prompt(messages),
    "multi_modal_data": {"image": [image_path]},
}
print(generate(inputs, limit_mm_per_prompt={"image": 1}))

Audio

audio_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/audio.mp3"
messages = [{
    "role": "user",
    "content": [
        {"type": "audio", "audio": audio_path},
        {"type": "text", "text": "Transcribe and summarize this audio."},
    ],
}]
inputs = {
    "prompt": build_prompt(messages),
    "multi_modal_data": {"audio": process_audio_info(messages, use_audio_in_video=False)},
}
print(generate(inputs, limit_mm_per_prompt={"audio": 1}))

Video with audio

video_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/vocal_video.mp4"
messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": video_path},
        {"type": "text", "text": "Describe the video, including relevant speech and sounds."},
    ],
}]
inputs = {
    "prompt": build_prompt(messages),
    "multi_modal_data": {
        "video": [video_path],
        "audio": process_audio_info(messages, use_audio_in_video=True),
    },
    "mm_processor_kwargs": {"videos_kwargs": {"fps": 1.0, "use_audio_in_video": True, "return_metadata": True}},
}
print(generate(inputs, limit_mm_per_prompt={"video": 1, "audio": 1}))

Video without audio

video_path = "https://raw.githubusercontent.com/TaoLiveAIGC/TLive-Omni/main/data/silence_video.mp4"
messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": video_path},
        {"type": "text", "text": "Describe the visual events in this video."},
    ],
}]
inputs = {
    "prompt": build_prompt(messages),
    "multi_modal_data": {"video": [video_path]},
    "mm_processor_kwargs": {"videos_kwargs": {"fps": 1.0, "use_audio_in_video": False, "return_metadata": True}},
}
print(generate(inputs, limit_mm_per_prompt={"video": 1}))

📖 Citation

If you find our work helpful, please consider citing our paper:

@article{hu2026tliveomni,
  title={TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming},
  author={Hu, Yibo and Qian, Yu and Gu, Mao and Tao, Yingfan and Chen, Yuhao and Luo, Yongdong and Liu, Zhuoqun and Jin, Meiguang and Ma, Junfeng},
  journal={arXiv preprint arXiv:2608.20958},
  year={2026}
}

📄 License

This project is released under the Apache License 2.0.