MOSS-VL-Realtime-SGLANG
English | 简体中文
MOSS-VL-Realtime-SGLANG provides the MOSS-VL realtime checkpoint with Transformers 5.12.1-compatible model and processor code for the specialized SGLang-Omni backend. This is a compatibility package, not a retrained or quantized model.
Features
- Continuous frame-by-frame video understanding and incremental text generation.
- Questions at any point in a stream, with response interruption.
- Proactive silence when no response is needed or evidence is insufficient.
- Updated responses as new visual information arrives.
- Timestamp-aware reasoning about event order and duration.
Quick Start
| Goal | Entry |
|---|---|
| Browser video/voice interaction and optional memory | MOSS-VL-Realtime Demo |
| Standalone streaming inference service | sglang-omni-realtime |
| Direct Transformers inference | Python examples |
This repository contains weights, configuration, tokenizer, processor, and custom code. It does not include a running service. The model is public; download the complete repository rather than individual weight files.
ASR, TTS, and memory are provided by the Demo. The model's offline Python APIs do not mean that the Demo's realtime deployment enables offline chat.
Demo memory is session-scoped: reconnects within the grace period retain it; final closure clears retrieval data. Saved conversation archives are separate. Use the named repositories' current code and complete model snapshots; installed revisions are diagnostic records, not fixed installation requirements.
Architecture and Configuration
MOSS-VL separates visual encoding from language reasoning with cross-attention. Timestamped frames and Cross-attention Rotary Position Embedding (XRoPE) align text and visual patches across time, height, and width.
| Item | Value |
|---|---|
| Parameters | 11B |
| Weights | BF16 |
| Model context | 256K |
| Vision patch size / temporal patch size | 16 / 1 |
| Default video preprocessing FPS / maximum sampled frames | 1.0 / 256 (offline video sampling, not a lifetime streaming-frame limit) |
| Direct Transformers runtime | One active realtime session per model instance |
| SGLang-Omni runtime | Configurable session capacity, subject to GPU memory |
The model context is not a per-session memory allocation guarantee. The recommended service configuration uses 131072 context; adjust context and concurrency for the available GPU memory.
Performance
MOSS-VL-Realtime targets streaming understanding, proactive silence, and dynamic response updates. Benchmark results are summarized below; see the technical report for model research.
Compatibility
Use this package's custom code and configuration together with the specialized backend. Installing a generic sglang-omni package does not supply the MOSS-VL integration.
The backend uses Transformers 5.12.1, SGLang 0.5.16, and PyTorch 2.11.0. This package adapts configuration, RoPE, and generation interfaces to Transformers 5.12.1 while keeping the original five BF16 weight shards, tokenizer, and vocabulary unchanged.
- Cross-attention Query RoPE: rotate each newly computed text query, including when visual KV is reused; cached visual keys are not rotated again. The corresponding fix is also available in the original reference implementation.
- Vision rotary frequencies: reconstruct canonical FP32 frequencies on the active device to handle buffer rematerialization in Transformers 5.12.1. This does not imply the same loading issue exists in the original 4.57 environment.
The original MOSS-VL-Realtime uses separate Transformers 4.57-series code. Do not mix its custom files into this package. Select matching model and backend versions; CUDA compatibility does not imply NPU support.
Limitations
- Response latency depends on hardware, frame rate, transport, and decoding settings.
- The direct Python runtime and the SGLang-Omni service have different session and backpressure behavior.
- Frame dropping and visual KV windows limit the visible history. Continuing beyond the context limit requires application-level memory handling.
- Applications should handle control tokens such as
<|silence|>,<|round_start|>, and<|round_end|>.
Related Models
| Model | Purpose |
|---|---|
| MOSS-VL-Realtime | Original realtime checkpoint and reference implementation |
| MOSS-VL-Instruct | Offline multimodal instruction following |
| MOSS-VL-Base | Continued pretraining and fine-tuning |
License
Apache-2.0. See OpenMOSS/MOSS-VL for the model family.
Citation
@misc{mossvl,
title = {MOSS-VL Technical Report},
author = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},
year = {2026},
eprint = {2608.15045},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2608.15045}
}
@misc{mossvideopreview,
title = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
author = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},
year = {2026},
eprint = {2606.07639},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2606.07639}
}