laya-omni (formal-v3)
Laya answers typed decisions (choose one option, give a score, answer yes/no) with calibrated probabilities. laya-omni adds optional image and audio inputs. Without an image or audio clip, the output is bit-identical to Laya.
- Code, docs, fine-tuning tool: https://github.com/ZHEQIUSHUI/laya-omni
- This repository: the trained fusion (about 7M parameters, 28 MB) and the
Qwen3-ASR-0.6B audio encoder extracted from the full ASR model (
audio_encoder/). - Frozen models it runs on: Laya multilingual (mmBERT-base, ~310M), SigLIP 2 base/16-256 (~93M), Qwen3-ASR audio encoder (186M).
Use
pip install "laya-omni @ git+https://github.com/ZHEQIUSHUI/laya-omni"
hf download zheqiushui/laya-omni --local-dir laya-omni
hf download convaiinnovations/laya-multilingual --local-dir laya-multilingual
from laya_omni import Omni
omni = Omni.load("laya-omni", laya="laya-multilingual",
image_encoder="google/siglip2-base-patch16-256",
audio_encoder="laya-omni/audio_encoder")
omni.predict("Image.", {
"sport": {"type": "choice", "instructions": "What game are they playing?",
"criteria": ["baseball", "tennis", "soccer"]},
"crowd": {"type": "noul", "instructions": "Is there an audience?"},
}, image="photo.jpg")
omni.predict("Audio clip.", {
"intent": {"type": "choice", "instructions": "What does the caller want to do?",
"criteria": ["check the balance", "freeze the card", "pay a bill"]},
}, audio="call.wav")
Several images: image=[a, b]. Images use 256 tokens by default (detail=False:
64). At load time the LoRA deltas are folded into a copy of Laya's encoder for
image / audio requests (same outputs, about a quarter faster; merge=False to
skip). One question with an image takes 14-20 ms on an RTX 4090, encoders included.
The previous release is kept under the tag formal-v2
(hf download zheqiushui/laya-omni --revision formal-v2).
Results
Standard benchmarks, same questions, one RTX 4090, one question at a time from the image or audio file to the answer (accuracy / median latency):
| Benchmark | laya-omni | Qwen3.5-0.8B | Qwen3.5-2B | Valen-Preview-0923 | Qwen2.5-Omni-3B |
|---|---|---|---|---|---|
| MMBench-EN dev | 0.66 / 14 ms | 0.75 / 42 ms | 0.84 / 46 ms | 0.73 / 70 ms | – |
| MMBench-CN dev | 0.60 / 14 ms | 0.75 / 42 ms | 0.82 / 45 ms | 0.73 / 70 ms | – |
| MMStar | 0.37 / 14 ms | 0.46 / 42 ms | 0.53 / 46 ms | 0.44 / 70 ms | – |
| MME (yes / no) | 0.68 / 17 ms | 0.76 / 48 ms | 0.82 / 62 ms | 0.61 / 105 ms | – |
| POPE | 0.84 / 16 ms | 0.88 / 44 ms | 0.90 / 51 ms | 0.82 / 88 ms | – |
| SEED-Bench (image) | 0.57 / 20 ms | 0.72 / 61 ms | 0.77 / 82 ms | 0.66 / 222 ms | – |
| OmniBench (image + audio) | 0.38 / 50 ms | – | – | – | 0.42 / 321 ms |
| MMAU test-mini (audio) | 0.45 / 20 ms | – | – | – | 0.67 / 91 ms |
About three times as fast as the smallest Qwen3.5 and 4-15 points less accurate on the image benchmarks; Qwen2.5-Omni-3B is well ahead on audio. On held-out COCO images it tells where an object is (0.85, 4 options), left / right / above (0.85), which is bigger (0.94) and which objects are present (0.96). Protocol, CircularEval, overlap checks and ablations: docs/results.md.
Adapting to a new scenario
A few hundred labelled images or clips, one folder per answer, one command: docs/finetune.md.
Limits
Weak on science diagrams (AI2D, TQA), medical images, spatial relations, and
questions about what happens next (game moves) rather than what is there.
Confident when wrong on unfamiliar diagram questions (TQA). Probabilities are
calibrated per modality and question type (temperature_by_modality in
fusion_config.json).
License
- Code: Apache-2.0 (repository).
fusion.safetensors: trained on public datasets that are not redistributed, plus questions generated from COCO 2017 boxes (CC BY 4.0) and option probabilities from Qwen3.8-27B (Apache-2.0); some datasets are for non-commercial research only (e.g. ScienceQA CC BY-NC-SA 4.0, ESC-50 CC BY-NC 3.0, Hateful Memes research license). Use it for research and non-commercial purposes; for commercial use, retrain with the repository's recipes on data whose licenses allow it.audio_encoder/: the audio encoder of Qwen/Qwen3-ASR-0.6B, unchanged, Apache-2.0 (license text and notice in that folder).- Runs on Laya (Convai Innovations) and SigLIP 2 (Google), both Apache-2.0, downloaded separately.