Laya-V

Typed decisions about images, in one forward pass, with calibrated probabilities.

Laya-V is the vision checkpoint of Laya. You give it an image (a photo, document, receipt, chart, screenshot or meme), optional text, and a set of typed questions: yes/no (noul), pick-one (choice) or rate-on-a-scale (score). It returns a probability for every option of every question at once, without generating text.

Highlights

• It reads the image, not just the text. 58.7% on held-out image questions, against 36.0% for an OCR + captioning pipeline (Florence-2-large) feeding the same text model and 32.0% with the image dropped. Laya-V beats the pipeline on 25 of 26 evaluation slices.
• Strong on everyday image domains: 87% on memes and posts, 87% on screenshots, 83% on documents and 81% on photos (held-out images, questions labelled by the teacher).
• 94% of its teacher's accuracy on an audited set (81.4% vs 86.8%), at a fraction of the latency, and level with it on documents and memes.
• Probabilities you can act on: ECE 0.024; keeping the 80% most confident answers lifts accuracy from 58.7% to 69.7% (documents: 87.6%).
• Many questions for almost the price of one: the image is encoded once, so 10 questions take 1.5x the time of one (133 vs 87 ms on a T4).
• Above its weight class: more accurate than Qwen3-VL-8B-Instruct (about 11x the parameters) on the same public-dataset questions, and better calibrated.

Quickstart

pip install "laya @ git+https://github.com/henilchopada/laya@laya-v" "transformers>=5.0"

import laya
from PIL import Image

router = laya.Router()  # text checkpoints load lazily; Laya-V loads on the first image request
result = router.predict(
    {"image": Image.open("receipt.jpg"), "note": "Reimbursement claim"},
    {
        "over_limit": {"type": "noul", "instructions": "Is the receipt total above 50?"},
        "category": {"type": "choice", "instructions": "What was bought?",
                     "criteria": {"meals": "food and drink", "travel": "transport, hotels",
                                  "other": "everything else"}},
        "legibility": {"type": "score", "instructions": "How readable is the receipt?",
                       "criteria": ["unreadable", "partly readable", "clear"]},
    },
)
answers = result["answers"]
answers["over_limit"]["noul"]            # P(yes), e.g. 0.93
answers["category"]["choice"]            # e.g. "meals"
answers["category"]["probabilities"]     # e.g. {"meals": 0.88, "travel": 0.07, "other": 0.05}
answers["legibility"]["score"]           # expected level on the 0..2 scale
answers["category"]["answer_confidence"] # calibrated confidence, use it to route or abstain
result["routing"]["model"]               # "vision"

# many images, same questions: states share forward passes
results = router.predict_batch([{"state": {"image": p}, "questions": questions} for p in paths])

# trade detail for speed: low = 256 patches, mid = 576, high = 1,024
router.predict({"image": img, "image_detail": "low"}, questions)

state["image"] takes a PIL image, a file path, raw bytes or base64. Any other keys in the state are read as text alongside the image (a caption, an email body, form fields).

Over HTTP, laya-serve (and the MCP server) also accept an image URL:

LAYA_DEVICE=cuda laya-serve
curl -s localhost:8000/predict -H 'content-type: application/json' -d '{
  "state": {"image": "https://example.com/receipt.jpg"},
  "questions": {"paid_by_card": {"type": "noul", "instructions": "Was it paid by card?"}}
}'

What it is for

Laya-V is built for closed-set decisions over images at high volume, where you need a probability rather than a paragraph:

• Expense and receipt checks: totals over a limit, payment method, tax shown, item present.
• Document triage: document type, which team should handle it, whether a field or signature is present.
• Content moderation: memes, posts and screenshots against your own policy categories.
• Screenshot and UI triage: what app or screen it is, whether an error is shown, what the user is doing.
• Photo tagging: scene, objects, attributes, quality.

Write the questions and options for your task; no fine-tuning is needed. Use answer_confidence to send low-confidence cases to a person or a larger model.

Not what it is for: free-form answers or captions, transcribing text (OCR), and locating UI elements on a screen (see Limitations).

Results

All numbers are on held-out images: no training row uses an evaluation image.

The image makes the difference

Domain rows: held-out images with teacher-labelled questions. Last row: every held-out image question, public datasets included. The OCR + caption pipeline gives the base laya-multilingual model a Florence-2-large caption and OCR text instead of the image.

Close to its teacher

An audited set of 280 questions whose reference answers were checked blind by an LLM judge (anthropic/claude-sonnet-5), not taken from the teacher:

Receipts and documents (public datasets)

Calibrated probabilities

Temperatures are fitted on a held-out slice and ship with the checkpoint, so the probabilities are usable as they come: when Laya-V says 80%, it is right about 80% of the time.

Rows cover every held-out question in the domain, public datasets included, so they differ from the teacher-labelled accuracies above. Screenshots are left out of this table because that slice is dominated by UI-element location (see Limitations).

Speed and cost

End-to-end laya.Router.predict on a Tesla T4 (fp16), image preprocessing included, 200 real evaluation images per setting.

At $0.35 per T4-hour that is about $0.0013 per 1,000 decisions (10 questions per image) or $0.0084 with one question per image.

Above its weight class

Both models answered the same 4,873 typed questions on the same held-out images, with the same options in the same order (Qwen3-VL-8B-Instruct through OpenRouter, asked for a probability per option). Accuracy is the mean over the public datasets (VQAv2, A-OKVQA, PixMo-Docs, CORD, ScreenSpot).

Laya-V is ahead with about a tenth of the parameters (paired McNemar test, p = 2e-06). Qwen's answers failed to parse on 6.5% of questions and those count as wrong, so part of the gap is formatting rather than vision.

Laya-V keeps the base model's text skills close to intact: 10 of 12 suites are within 1 point or better.

How it works

• Vision tower: SigLIP 2 So400m/16 NaFlex, which keeps each image's native aspect ratio (tall receipts and wide charts are not squashed). The patch budget is 256, 576 or 1,024 patches.
• Adapter: per-channel standardisation of the tower features, aspect-aware pooling of the patch grid to 128 tokens, a 2-layer MLP projector to width 768, and learned <img> / </img> vectors.
• One pass for all questions: each question becomes one row [CLS] <type> instructions [SEP] [MASK] option… [SEP] <img> 128 image tokens </img> text state [SEP]. The image is encoded once and its tokens are shared by every row; the mmBERT encoder and Laya's decision head score all options of all questions in one batched forward pass.
• Text-only requests skip the vision path entirely and behave like laya-multilingual.

Training

1. Alignment A1: image-conditioned masked language modelling on captions (COCO, PixMo-Docs, PixMo-Cap); adapter only.
2. Alignment A2: 4-way caption matching on clusters of 4 similar images, so text alone cannot solve it; text encoder trainable at a low learning rate, text replay mixed in.
3. Decisions B: Laya's RLCD loop (GRPO-style noisy logits rewarded by strictly proper scoring rules, plus soft cross-entropy) on teacher-labelled image questions, public datasets converted to typed questions, and self-distilled text replay; a short B2 top-up with broader text replay limits text drift. The released weights are a WiSE-FT merge: text encoder and decision head at 0.7 x fine-tuned + 0.3 x laya-multilingual.
4. Calibration C: temperatures fitted by LBFGS on a held-out slice, separately for text and image answers.

Data and licenses

Teacher labels come from Space Bunny Alpha, an OpenRouter stealth model, as stated probability distributions, with a shuffled-option consistency check on 15% of images. Every training row carries its source, so teacher rows can be replaced if the teacher's terms require it. Dataset licenses cover the annotations; the underlying images (COCO from Flickr, PixMo-Cap web images) carry their own terms.

Limitations

• Closed-set decisions only. Laya-V scores the options you give it; it does not generate answers. It is not a general visual question answering model, and large generative VLMs are more accurate on open-domain VQA benchmarks.
• Locating UI elements is weak. Questions like "where on the screen is the Save button?" are answered only modestly above chance; use a grounding model for that.
• Confidence does not drop on uninformative images. On a blank grey image the mean top probability is 58.5% (vs 58.3% on real images), so check that an image is present and readable before trusting a confident answer.
• Text-only drift: suites over the 1-point limit: prompt_injections, massive_ar. Use laya-multilingual for text-only prompt-injection filtering.
• One image per request. The fast path (fast=True), ONNX export and predict_long do not accept images yet.
• Teacher labels carry the teacher's errors and some of its overconfidence (partly corrected by calibration), and Laya's known quirks (negation errors, score position bias in some languages) apply here too.

Citation

@misc{laya_v_2026,
  title        = {Laya-V: calibrated typed decisions over images in one forward pass},
  author       = {Chopada, Henil},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/henilchopada/laya-v}}
}

Laya-V builds on Laya and convaiinnovations/laya, SigLIP 2 and mmBERT.