Youtu-Parsing-Omni

News

• [2026-10] OmniDocBench evaluation code released (see OmniDocBench Evaluation).
• [2026-10] Technical report, model weights, vLLM plugin and inference examples released.

Introduction

Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video — it produces one structured JSON envelope that covers both perception (layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, camera motion) and cognition (captions, narratives, reports). The output family is selected by the task prompt (--task in the examples, keys of prompts/youtu_parsing_omni.json).

Highlights (see the technical report for details):

• Unified schema – one JSON envelope for seven parsing families, driven by the task prompt.
• Omni encoder – image, audio and interleaved audio-visual video inputs (frames + audio track) in a single model.
• Strong results at a small size – state-of-the-art on OmniDocBench v1.6 (96.96 Overall), best open-weight model on OmniParsingBench (75.08 Avg., second only to Gemini-3-Pro), and competitive with specialized models on chemical-structure (ChemOCR) and music-score (PDMX-Synth) recognition (results).
• Easy to serve – a vLLM plugin, pinned serving settings, task prompts and inference examples are included.

Method

One encoder for every modality

Existing omni-modal parsers (a) encode pixels and audio with two separately pre-trained towers whose streams first meet inside the language model. Youtu-Parsing-Omni (b) replaces both towers with a single Youtu-Omni-Encoder, initialized from a pre-trained text language model to strengthen text reading. Thin modality-specific stems map pixels and log-mel frames to tokens; a shared bidirectional Transformer contextualizes an arbitrary packed mix of images, audio chunks and videos under one (t, h, w) positional encoding; modality-specific mergers and projectors then feed the tokens, together with the text prompt, to the Youtu-LLM decoder, which emits OmniSchema JSON. Only the stems and the merger / projector heads depend on the modality, so almost all perception parameters are shared.

Audio-visual fusion inside perception

The frames and audio chunks of a video are packed in temporal order. (a) A non-fusion layer gives every frame and every audio chunk its own attention window, while a few fusion layers start a new window at every audio-to-frame boundary, so a frame attends to the sound of the same instant. (b) Vision tokens take t from the frame index and (h, w) from the patch grid; audio tokens take t from their time offset with h = w = 0, so both modalities share one temporal coordinate system. Cross-modal alignment is thus computed inside the encoder rather than reconstructed by the decoder.

One schema for every task: OmniSchema-Aware On-Policy Distillation (OSAD)

All ten parsing tasks share OmniSchema, so after SFT and schema-routed RLVR a single self-evolving post-training procedure, OSAD, applies the same token-role-based objective across tasks. The student samples its own parses; rollouts that pass the admission gate (JSON validity, schema conformance, box syntax, repetition checks) are scored by a frozen copy of the model that additionally sees the reference parse. Because every output follows OmniSchema, each token is identified as content (Jensen–Shannon divergence) or structure (forward KL at a higher temperature), plus an all-token JS term. At every round boundary the student's LoRA adapters are merged and the student becomes the next teacher, so the parser and its supervision co-evolve without a hand-designed reward or a separately trained teacher.

Model Zoo

Results

OmniDocBench v1.6

OmniParsingBench

ChemOCR & PDMX-Synth

ChemOCR

PDMX-Synth

Quick Start

1. Installation

git clone https://github.com/TencentCloudADP/youtu-parsing.git
cd youtu-parsing/youtu_parsing_omni

# choose one mode per Python environment (Python >= 3.10, CUDA GPUs for inference)
bash scripts/setup_env.sh                 # vLLM serving + plugin (transformers==5.2.0)
bash scripts/setup_env.sh --transformers  # Transformers inference (transformers==5.10.2)

• vLLM serving and Transformers inference pin different transformers versions (5.2.0 and 5.10.2); install them into separate Python environments. peft must not be installed in the serving environment (the script removes it).
• ffmpeg must be available for audio / video inputs.

# serving (vllm==0.19.0, transformers==5.2.0)
pip install -r requirements/vllm.txt
pip install transformers==5.2.0   # overrides vllm's transformers<5 pin
pip uninstall -y peft
pip install --no-deps ./vllm-plugin-vita-omni
# Transformers inference (separate env, transformers==5.10.2)
pip install -r requirements/transformers.txt

2. Serve with vLLM

MODEL=tencent/Youtu-Parsing-Omni DATA_ROOT=/path/to/media GPU=0 bash scripts/vllm.sh

This starts an OpenAI-compatible server on http://127.0.0.1:8000/v1 (served model name Youtu-Parsing-Omni) with the vllm-plugin-vita-omni model implementation and the runtime settings of the reported results.

• Engine settings: --max-model-len 65536 --max-num-batched-tokens 65536 --max-num-seqs 1, --gpu-memory-utilization 0.70, --limit-mm-per-prompt '{"image": 8, "video": 4, "audio": 8}', --seed 1234, bfloat16; vLLM defaults for everything else.
• The server listens on 127.0.0.1 only and runs in the foreground (logs go to the terminal); Ctrl-C stops it. vLLM's internal port is 30000 (override with VLLM_PORT).
• The script also exports the media token budgets read by the plugin (YOUTU_VITA_*, see vllm-plugin-vita-omni).
• MODEL is passed to vllm serve as is: tencent/Youtu-Parsing-Omni is downloaded from the Hugging Face Hub (cached in HF_HOME); a local checkpoint directory works as well.

3. Parse a file

The media file must lie below DATA_ROOT.

python examples/infer_vllm.py --task document           --media /path/to/media/page.png
python examples/infer_vllm.py --task natural_image      --media /path/to/media/photo.jpg
python examples/infer_vllm.py --task graphics_chart     --media /path/to/media/chart.png
python examples/infer_vllm.py --task graphics_flowchart --media /path/to/media/flowchart.png
python examples/infer_vllm.py --task graphics_geometric --media /path/to/media/geometry.png
python examples/infer_vllm.py --task audio              --media /path/to/media/speech.wav
python examples/infer_vllm.py --task natural_video      --media /path/to/media/clip.mp4
python examples/infer_vllm.py --task textrich_video     --media /path/to/media/lecture.mp4

Use --api-base to target another server. The task prompts are listed in prompts/youtu_parsing_omni.json. Requests must put the media before the text and enable the parsing mode of the chat template shipped with the model (chat_template_kwargs = {"enable_thinking": false, "enable_parsing": true}), which renders the media token directly followed by the prompt, without role markers. As in the reported results, decoding is greedy, stops at <|image_pad|> / <|end_of_text|> (stop_token_ids, looked up through the server's /tokenize endpoint) and keeps special tokens, which carry the coordinates.

import requests
from openai import OpenAI

# token ids of <|image_pad|> and <|end_of_text|>
stop_token_ids = sorted(requests.post("http://127.0.0.1:8000/tokenize", timeout=60, json={
    "model": "Youtu-Parsing-Omni", "prompt": "<|image_pad|><|end_of_text|>",
    "add_special_tokens": False}).json()["tokens"])

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
prompt = "Please perform end-to-end document parsing on the provided image. ..."  # see prompts json
response = client.chat.completions.create(
    model="Youtu-Parsing-Omni",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "file:///path/to/media/page.png"}},
        {"type": "text", "text": prompt},
    ]}],
    temperature=0.0,
    max_tokens=32768,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False, "enable_parsing": True},
        "skip_special_tokens": False,
        "min_tokens": 16,
        "stop_token_ids": stop_token_ids,
    },
)
print(response.choices[0].message.content)

Audio files are sent as audio_url. For videos send one file:// video_url below --allowed-local-media-path and add "mm_processor_kwargs": {"nframes": 64, "use_audio_in_video": true, "use_vision_in_video": true} to extra_body; the plugin then decodes the audio track together with the frames (see Video requests).

Transformers

Transformers inference requires transformers==5.10.2 (bash scripts/setup_env.sh --transformers, in an environment separate from vLLM). A minimal example is in examples/infer_transformers.py:

MODEL=tencent/Youtu-Parsing-Omni

python examples/infer_transformers.py --model $MODEL --task document           --media /path/to/media/page.png
python examples/infer_transformers.py --model $MODEL --task natural_image      --media /path/to/media/photo.jpg
python examples/infer_transformers.py --model $MODEL --task graphics_chart     --media /path/to/media/chart.png
python examples/infer_transformers.py --model $MODEL --task graphics_flowchart --media /path/to/media/flowchart.png
python examples/infer_transformers.py --model $MODEL --task graphics_geometric --media /path/to/media/geometry.png
python examples/infer_transformers.py --model $MODEL --task audio              --media /path/to/media/speech.wav
python examples/infer_transformers.py --model $MODEL --task natural_video      --media /path/to/media/clip.mp4
python examples/infer_transformers.py --model $MODEL --task textrich_video     --media /path/to/media/lecture.mp4

--task takes the same values as examples/infer_vllm.py and selects the prompt from prompts/youtu_parsing_omni.json; the media path is not restricted to a DATA_ROOT here. Hub ids are downloaded to --download-dir (default ~/.cache/youtu-parsing-omni). The model is loaded with trust_remote_code=True, which executes the Python code shipped with the checkpoint; when --model is a Hugging Face Hub id, pin a reviewed commit with --revision <commit-hash>. The reported results were produced with vLLM (scripts/vllm.sh).

Output format

The model answers with one fenced ```json block. Coordinates are encoded with dedicated tokens relative to a 0–1000 grid, e.g. "bbox": "<box><x_85><y_275><x_1000><y_1000></box>" (top-left x/y, bottom-right x/y); timestamps use HH:MM:SS. The top-level modality / subtype pair of each task is listed in the Introduction; image tasks return elements (plus reading_order for document-style outputs), audio and video tasks return temporal segments, and every output has a global_description.

{
  "modality": "image",
  "subtype": "natural_image",
  "elements": [
    {"id": "e_000", "category": "entity", "bbox": "<box><x_350><y_465><x_590><y_715></box>",
     "tag": "snake bookmark", "content": {"text": "", "caption": "A silver snake-shaped bookmark ..."}}
  ],
  "global_description": {"caption": "...", "narrative": "...", "scene": "..."}
}

OmniDocBench Evaluation

Inference runs in the vLLM environment; scoring uses the official OmniDocBench evaluator via Docker (ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204).

MODEL=tencent/Youtu-Parsing-Omni OMNIDOCBENCH=/path/to/OmniDocBench \
OUTPUT_DIR=outputs/omnidocbench_$(date +%Y%m%d_%H%M%S) bash scripts/eval_omnidocbench.sh

• OMNIDOCBENCH: dataset directory containing images/ and OmniDocBench.json.
• GPUS: GPUs to use (default: all visible), one vLLM server per GPU.
• To resume, rerun with the existing OUTPUT_DIR path (not a new timestamp).
• Outputs: predictions in <OUTPUT_DIR>/predictions/, final report in <OUTPUT_DIR>/results.md, logs in <OUTPUT_DIR>/logs/.

## Citation

If you find Youtu-Parsing-Omni useful in your research or applications, please consider citing our work:

```bibtex
@article{youtu-parsing-omni,
  title   = {Youtu-Parsing-Omni: One Encoder, One Schema, Every Modality},
  author={Tencent Youtu Lab},
  journal = {arXiv preprint arXiv:[TBD]},
  year    = {2026}
}

@article{youtu-parsing,
  title={Youtu-Parsing: Perception, Structuring and Recognition via High-Parallelism Decoding},
  author={Tencent Youtu Lab},
  year={2026},
  eprint={2601.20430},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2601.20430},
}

@article{youtu-vl,
  title={Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision},
  author={Tencent Youtu Lab},
  year={2026},
  eprint={2601.19798},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2601.19798},
}

@article{youtu-llm,
  title={Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models},
  author={Tencent Youtu Lab},
  year={2025},
  eprint={2512.24618},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2512.24618},
}

Acknowledgements

We extend our gratitude to the following projects and communities that made Youtu-Parsing-Omni possible:

• Youtu-VL
• Youtu-LLM