Rune 26B-A4B v3

Rune is a decision model for text and images with a context size of 262k: give it a state, which can be text, structured data or a picture with text, plus a question and a set of options, and it returns one option with a probability over all of them, in a single forward pass. It answers three kinds of questions:

• choice (pick one of N)
• noul (a probability that a statement is true)
• score (a position on an ordinal scale)

This is Rune v3 as bfloat16 safetensors, a standard Gemma4ForConditionalGeneration checkpoint (26.5B parameters, 8 of 128 experts active per token). It is served fastest by the surogate engine, whose decisions endpoint implements the protocol Rune was trained with. Every number on this page was measured, and the harnesses are named, so you can reproduce them.

The Rune v1 GGUF builds that this repository used to hold are still in its history, at revision 2a15504.

Accuracy

Decision Index is a public leaderboard of decision models. Its current edition, 0.2.1, scores the 0.2 suite (151,034 requests from 44 benchmarks) on a 38-benchmark panel: each benchmark's accuracy is rescaled so that chance scores 0 ("skill"), averaged within five areas, and the areas are weighted into the headline index. Rune v3's row is the leaderboard's: our submitted run of these bf16 weights, adopted after the maintainers' own reproduction matched it bit for bit. Index and area skill scores in %, as published on 26 September 2026:

* Our estimate, not yet on the leaderboard: the leaderboard's per-benchmark 0.2.1 scores for Rune v3, each moved by the change thinking made to that benchmark in our full run of the suite with the official 0.2 scorer (the 0.2.1 scorer is not public yet). Rebuilt the same way, the published rows for Rune v3 (57.44) and Jev 1.13 (57.89) come out exactly.

The index counts unanswered requests as wrong. Rune v3 is second on the board, 0.45 points behind Jev 1.13, and with thinking on it would lead it by about 1.3. On the previous edition, 0.2 (40 benchmarks, a plain mean of the areas), Rune v3 scored 53.39, and 54.89 with thinking, against 51.67 for Jev 1.13. 0.2.1 takes RouterBench and SGD out of the panel, weighs Knowledge & Reasoning and Language at 26% each and Arts & Human Taste at 10%, and rescores five benchmarks.

Calibration

Measured on the Decision Index's 33 benchmarks that have a right or wrong answer per field (each benchmark weighted equally, every request). At the default temperature Rune is overconfident; reading the answers at temperature 2 makes the probabilities match accuracy:

The temperature changes only the probabilities, never the chosen option. With surogate, pass --decision-temperature 2 to surogate serve.

Thinking

Thinking is opt-in, per request. With "thinking": true, a question Rune is unsure of thinks before it answers: if its one-pass confidence (the top option's probability; for a true/false question the larger of p and 1 - p) is below 0.7, Rune reasons for up to 512 tokens and then answers; every other question keeps its one-pass answer.

On the Decision Index 0.2 suite (our full run, scored with the official 0.2 scorer), about one question in ten thinks and the 0.2 index rises from 53.39 to 54.89, with Knowledge & Reasoning from 43.5 to 48.4; under 0.2.1's weights that is about 59.2 against 57.44 (see Accuracy). The largest gains in skill points: GSM8K +15.2, CRUXEval +11.4, CLadder +10.4, BBH +8.0, PhishNChips +5.1. ForecastBench, whose answers are probabilities, loses 9.8: a thought makes those probabilities more extreme.

A question that thinks takes about 5 seconds instead of 0.2 (see Speed); the others are unchanged. Thinking is not available together with images yet.

Images

The vision tower is kept, so Rune accepts images. With surogate, start the server with --vision and put the picture in the request's images array as a data URL; it is placed ahead of the text.

We measured image decisions with the item format and scoring of Image JevBench v0.1 on 1,196 items: the benchmark's 136 public items (its 8 example cards, and its 128 preview items rebuilt from their original sources) and 1,060 items we built the same way from the same openly licensed sources. This is our own run, not a leaderboard result: the benchmark's sealed items are private and were not used, so our items are a different sample. Accuracy in %, one request at a time on one RTX PRO 6000 with surogate 1.5.3, every item answered:

The screen families ask which of five labelled markers drawn on the screenshot to click. The benchmark's own everyday photos are synthetic and mostly sealed, so that row is 100 real VizWiz photos plus the benchmark's 2 example photos; ArxivQA's answer labels were machine-generated.

Image probabilities are well calibrated at the default decision temperature (ECE, ten bins, top option):

The --decision-temperature 2 recommended above for text keeps image ECE within 1.5 points of temperature 1.

Image token budget. Gemma 4 turns each image into at most 280 tokens by default. Raising the image processor's budget to 1,120 (max_soft_tokens and image_seq_length in processor_config.json) gives Rune more detail: +6.3 points over all items (paired 95% interval +4.3 to +8.4), most of it on small screen targets (ScreenSpot-Pro +20.8) and dense pages and tables (Mind2Web +14.7, FinQA +10.7). It costs time: one image decision took 0.18 s at 280 tokens and 0.46 s at 1,120 (median, one request at a time, RTX PRO 6000).

Speed

Measured during the Decision Index run above: one NVIDIA RTX PRO 6000 Blackwell per server, bf16, surogate 1.5.3 with the MoE routing fix described below, four concurrent clients per GPU. A Decision Index request carries a shared state and often many questions, so these figures are per request, not per question:

One question per request, one request at a time, on an idle RTX PRO 6000 (surogate main):

With thinking on, questions at or above the confidence gate answer as fast as without it. On the Decision Index's mix, where one question in ten thinks, that averages about 0.65 s per question.

Running it

Thinking is off, and that is the default: the chat template in these files disables it.

With surogate (recommended)

The weights take 51.6 GB, so they need a GPU with room for them plus the KV cache; we serve them on 96 GB RTX PRO 6000 cards:

hf download surogate/rune-26b-a4b-GGUF --local-dir rune-v3
surogate serve ./rune-v3 --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 --max-num-seqs 8 --kv-capacity auto --decision-temperature 2

Under sustained concurrent load, surogate 1.5.3 can occasionally hit a non-finite router value in the MoE layers and stop with an illegal memory access. surogate's main branch includes a fix (#217): with it, an affected request fails cleanly and the server keeps running. Use a build that includes it.

Then ask it a decision (use the model id that GET /v1/models lists; add "thinking": true next to "questions" to let unsure questions think, which needs a surogate build that includes #240):

curl http://localhost:8000/api/alpha/decisions -H 'Content-Type: application/json' -d '{
  "model": "<id from /v1/models>",
  "state": "Customer wrote: the package arrived late and damaged, I want my money back.",
  "questions": {
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?",
               "criteria": {"true": "A refund is requested", "false": "No refund is requested"}}
  }
}'

With transformers

The checkpoint loads like any Gemma 4 checkpoint. A decision is read at the first generated position as a softmax over the option letters' logits, with the prompt rendered as surogate's decisions endpoint renders it (docs/inference/decisions.md).

There are no GGUF builds of v3 yet.

Files