Rainier-VL-2B-MTP

Paper: Rainier-VL: A Compact Vision-Language Model for Industrial Inspection Across Edge GPUs Myan Sudharsanan, Hemmanth Puppala, Gavin Daher, and Jack Rosenbloom Denali Advanced Integration Physical AI Team

Abstract

Efficiently running defect detection via visual inspection on a production line or on returns benches is a problem that is the intersection of three concerns, speed, accuracy, and cost. In the domain of defect detection, no other Vision Language Model (VLM) can satisfy the constraints for the areas of concern out of the box. Hence, we present Rainier-VL, a 2.1 B-parameter VLM that meets that operating point across the edge GPU range, from a 50 W, 8 GB workstation card to a Blackwell-class part. There are two novel architectural advancements with Rainier-VL. The OrthoBernstein projector treats the vision→language connector as Householder-orthogonal layers composed with a learnable Bernstein activation, and an SVD warm-started to a dense teacher. This results in giving 24.2× fewer parameters and 12.2× fewer forward FLOPs than the dense equivalent. A hybrid state-space backbone replaces the growing KV cache with a fixed recurrent state, making per-frame cost constant on a continuous camera feed. Rainier-VL reaches 0.949 image-AUROC on VisA defect detection, ahead of every open-source 2–10 B model we evaluate in the same harness, including a 10.3 B model at 0.852, at 4.6× less inference memory than that model and no quantization anywhere.

This repository

This is the Rainier-VL-2B checkpoint with multi-token prediction (MTP), packaged with its own inference engine and an OpenAI-compatible server. One image plus a short instruction returns a defect verdict with a calibrated probability, per-defect segmentation masks, bounding boxes, a dense defect-probability map, or a one-sentence description. It runs on an 8 GB GPU such as the RTX A1000.

Model details

One vision forward per image produces a shared projected grid that the language model, the box head and the mask heads all read, so asking for a verdict, a box and a mask on the same frame pays the vision cost once.

Tasks

The task is selected by a tag at the start of the prompt.

For detect the engine fills in the canonical question itself; send only the tag. od and anomaly reuse the seg pass on the same frame, so asking for all three costs about one seg call.

Benchmarks

Defect detection vs open-source VLMs

Image-AUROC across four domains. Every row is scored in the same harness on the same images (per-category cap 40). Parameters are unique parameters.

Logical anomalies (MVTec-LOCO) are excluded: every model evaluated, Rainier-VL included, scores between 0.47 and 0.56 there. The strongest peer on VisA (GLM-4.6V-Flash) needs 20.9 GB of GPU memory; Rainier-VL peaks at 6.4 GB.

Full held-out reserves

On the complete held-out reserves (no per-category cap) this checkpoint reaches:

Segmentation heads

Validation IoU of the mask heads shipped in weights/heads/:

Decode cost vs open-source VLMs

Per-token decode cost on an RTX PRO 6000, best of 3 after three warm-ups. Rainier-VL is measured with the bundled rainier_engine (static hybrid cache, fused multi-token verify, MTP speculative decoding); the other models are served through vLLM with CUDA graphs.

Speed and latency

RTX A1000 8 GB (50 W, deployment target)

Measured by the bundled engine per request, single stream, one 384 px frame, fresh compute (caches excluded), card under sustained load.

Prefill of a 739-token sequence takes about 390 ms at the default chunk_size=256. Peak GPU memory is about 6.4 GB with MTP and all heads loaded.

In a deployed inspection station (capture, inference, overlay and write, 2,747 frames over 65 minutes, two camera streams) the full application runs at 908 ms per frame end to end, 1.101 frames/s sustained, with per-camera revisit at 1,591 ms (p10 1,504 ms, p90 1,669 ms). Adding mask and box output to a verdict-only frame costs 39 ms.

RTX PRO 6000 (Blackwell)

The engine's cost per request is a fixed part (preprocess, vision encode, prefill) plus a per-token part, so answer length is the biggest lever you control.

Fixed cost ≈ 55.6 ms plus ≈ 3.40 ms per generated token. Batching two frames in one call amortises about 57% of the per-request overhead.

Vision encoder alone, batch 1 / 4 / 8: 19.9 / 36.5 / 70.3 ms.

Throughput

On a static scene the server answers repeats from its scene cache, so a monitoring loop runs far above the fresh-compute rate and pays the fresh cost once when the scene changes.

Multi-token prediction (MTP)

MTP adds four small heads to the language model, each predicting one of the next four tokens from the model's last hidden state. They are trained jointly with the base model and share its embedding table, so they add only 16.8 M parameters. At inference MTP proposes several tokens at once and the base model checks them all in a single forward pass, keeping only the tokens that match its own greedy choice. Output is identical to plain greedy decoding, and each forward pass commits more than three tokens on average.

Determinism

Outputs are bit-identical within a process and across server restarts when CUBLAS_WORKSPACE_CONFIG=:4096:8 and a persistent TRITON_CACHE_DIR are set before launch.

Requirements

• NVIDIA GPU with 8 GB VRAM or more, Ampere or newer. Peak use is about 6.4 GB. The 4 GB RTX A1000 variant cannot run this model, and there is no quantized version.
• CUDA 12.8-capable driver, Python 3.12
• torch==2.10.0, transformers==5.5.4 (pinned; other versions change the SigLIP module layout), mamba-ssm and causal-conv1d CUDA wheels. See requirements.txt.
• For the HTTP server additionally: fastapi, uvicorn, scipy.
• Access to google/siglip-so400m-patch14-384 and Zyphra/Zamba2-1.2B on the Hugging Face Hub. The loader builds both towers from these configs and then loads Rainier's weights over them. After the first download they are read from the local HF cache.

Quick start

1. Download and install

hf download Denali-AI/Rainier-VL-2B-MTP --local-dir rainier-vl-2b-mtp
cd rainier-vl-2b-mtp

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install fastapi uvicorn scipy

2. Check the GPU and run the smoke test

PYTHONPATH=code python -m ortho_vlm.kernel_probe   # reports which decode kernel tier this GPU gets
python smoke_test.py                               # loads the weights and checks for a valid JSON answer

smoke_test.py must end with PASS. It verifies that the weights and MTP heads load and that the model returns a parseable {"defect": ...} answer.

3. Python API

import sys; sys.path.insert(0, "code")
from PIL import Image
from rainier_engine import Rainier

m = Rainier.from_pretrained("weights/vlm.pt", drafter="weights/drafter/head.pt")
img = Image.open("part.jpg").convert("RGB")
out = m.generate(img, "Describe any defect in this image.", max_new_tokens=64)
print(out["texts"][0])

The Python API covers generation (detect and caption). The mask heads, object profiles, confidence gate and scene cache live in the server.

4. Start the server

export CUBLAS_WORKSPACE_CONFIG=:4096:8          # deterministic outputs across restarts
export TRITON_CACHE_DIR=$PWD/.triton-cache      # persistent compiled kernels
python serve/engine_shim.py --model-dir . --port 8150

The server warms up before it starts listening; wait for ready on :8150. Then:

curl localhost:8150/health       # lists loaded modes and spec-decode status

For an offline host, copy a populated HF cache alongside the model and use serve/serve_standalone.sh (it sets HF_HUB_OFFLINE=1); set MODEL_PATH to this directory and PYTHON to your venv's interpreter.

5. Send a request

import base64, io, json, requests
from PIL import Image

img = Image.open("part.jpg").convert("RGB")
img.thumbnail((384, 384), Image.Resampling.BOX)       # longest side 384, never upscale
buf = io.BytesIO(); img.save(buf, "JPEG", quality=90)
uri = "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode()

def ask(text, max_tokens=24):
    r = requests.post("http://localhost:8150/v1/chat/completions", json={
        "model": "rainier-vl-2b-jointmtp-v2-a1000",
        "temperature": 0, "max_tokens": max_tokens,
        "messages": [{"role": "user", "content": [
            {"type": "text", "text": text},
            {"type": "image_url", "image_url": {"url": uri}}]}]})
    return r.json()["choices"][0]["message"]["content"]

verdict = json.loads(ask("[detect] [pf:metal]"))
print(verdict["answer"], verdict["p_yes"])

seg = json.loads(ask("[seg] [pf:metal] Segment the defect."))
for inst in seg.get("instances", []):
    print(inst.get("label"), inst.get("severity"), inst["bbox"])

print(ask("[caption] Describe this metal part in one short sentence, "
          "noting any visible damage or defects.", max_tokens=64))

detect, seg, od and anomaly return a JSON string in message.content; caption returns plain text. Boxes are normalized [x0, y0, x1, y1]. Masks are run-length encoded as {"size": [h, w], "counts": [...]}, row-major, alternating 0-runs and 1-runs starting with a 0-run. "stream": true is supported and returns the whole answer as one SSE chunk.

Prompting rules

• Send the bare prompt. Do not wrap it in USER: / ASSISTANT: or any chat-role template. The model was trained on bare prompts and a role wrapper breaks the answer format.
• Put tags first, then the task text. Tags are stripped before the text reaches the model.
• Use temperature: 0.

Control tags

Slider values of 0.5 are neutral.

Object profiles

A profile tells the model what kind of object is in frame and which defect types to look for.

The full registry, including the probe questions used per defect type, is in hyperparameters/object_profiles.engine.json. New profiles are added to PROFILES in serve/engine_shim.py.

Calibrating for a new line

The generated yes/no verdict should be calibrated per camera before it is trusted on a new production line. Collect about 20 clean and 20 defective frames per camera, measure how well p_yes separates them, and write the per-camera thresholds to serve/detect_calib.json (Denali's calibrate_detect.py tool produces this file). With [gate:on] the server then applies the threshold with a 3-frame rolling median per camera, and masks follow the gated decision. Enable the gate only on cameras where calibration AUROC is 0.9 or higher.

No calibration file ships with this repository. Without one, [gate:on] has no effect and the server returns the model's generated verdict.

Configuration

Server behaviour can be tuned through environment variables on the serve process. Defaults and descriptions are in hyperparameters/engine_env_defaults.json. The most commonly changed:

Limitations

• Defects smaller than about 10 px at 384 px are usually detected but not localized. Use a closer crop or a zoomed camera for them.
• The scene cache matches frames with a perceptual hash. On a static view, a small new defect can be served the previous clean result for up to five requests; send [fresh] when every frame must be recomputed.
• The yes/no verdict is not calibrated for new lines out of the box (see above).
• Speculative decoding speeds up the short JSON answers of detect; free-form captions fall back to greedy decoding.
• The server processes one request at a time on one GPU.

Files

Citation

@techreport{sudharsanan2026rainiervl,
  title       = {Rainier-VL: A Compact Vision-Language Model for Industrial Inspection Across Edge GPUs},
  author      = {Sudharsanan, Myan and Puppala, Hemmanth and Daher, Gavin and Rosenbloom, Jack},
  institution = {Denali Advanced Integration},
  year        = {2026},
  note        = {Model: https://huggingface.co/Denali-AI/Rainier-VL-2B-MTP}
}

License

Released under the Apache License 2.0. Copyright 2026 Denali AI.