OmniLion
OmniLion combines the SEA-LION/Qwen3.6 language and vision model with a Qwen3-ASR audio encoder and a trained audio projector.
A single native vLLM process supports:
- text → text
- image → text
- video → text
- audio → text
- joint video + audio → text
This is a Hugging Face SafeTensors model, not GGUF. The matching vLLM plugin is required.
Install
The verified runtime is Linux ARM64 on NVIDIA GB10 with Python 3.12, vLLM 0.29.0, PyTorch 2.13.0+cu129, and Transformers 5.17.0. Install the appropriate vLLM/PyTorch build for your platform first, then install the included plugin:
python -m pip install --no-deps ./omnilion_vllm_plugin-0.1.0-py3-none-any.whl
Serve
export VLLM_PLUGINS=omnilion
export PYTHONNOUSERSITE=1
vllm serve LLJYY/OmniLion \
--served-model-name OmniLion \
--host 127.0.0.1 \
--port 8002 \
--dtype bfloat16 \
--max-model-len 8192 \
--max-num-seqs 1 \
--gpu-memory-utilization 0.65 \
--limit-mm-per-prompt '{"image":1,"video":1,"audio":1}' \
--media-io-kwargs '{"video":{"num_frames":30}}' \
--chat-template-content-format string \
--enforce-eager
Use /v1/chat/completions with model: "OmniLion". OpenAI-compatible content
parts are image_url, video_url, and input_audio. For joint AV, place the
video and audio parts before the text instruction in the same user message.
Audio is limited to 30 seconds per item in this release.
Thinking
OmniLion keeps Qwen3.6's thinking behavior enabled for normal requests. Clients
can pass chat_template_kwargs: {"enable_thinking": false} for strict-format,
low-latency, or deterministic health-check calls.
Promotion status and benchmark proxy
This revision was promoted by explicit user override after the automatic evaluation gate failed. The failed gate remains immutable and is not represented as a pass. These values are retained as a benchmark proxy for future repair adjustments:
- MMAU: 664/1000, versus 669/1000 for the previous accepted revision
- Daily-Omni joint: 45/64, versus 46/64
- Tamil: 458/900, versus 475/900
- Silence/room tone: 32/32; cough: 8/8; sigh: 8/8; throat clearing: 7/8
- Partial/abandoned speech: 0/32
- Intelligible speech with inserted events incorrectly suppressed: 0/16
Known weak areas are partial/abandoned speech, one throat-clear case, and Tamil retention.
Notes
- The release was verified with text, image, video, audio, and joint-AV requests through one vLLM process and through LiteLLM.
- Only BF16 on the pinned GB10 runtime is release-verified.
- The model can hallucinate and inherits the safety and bias limitations of its components and subsequent training.
- Training data and evaluation media are not included.
- Keep raw vLLM private; put an authenticated gateway such as LiteLLM in front for networked deployment.
Exact files and hashes are in release-manifest.json. Lineage is in PROVENANCE.md, and component terms are in LICENSES.md.