QJev 3.5-0.8B: a small decision engine you can run on a laptop

https://huggingface.co/ZY-Lee4/QJev3.5-0.8B-GGUF/resolve/main/qjev_demo.mp4

Fifteen seconds of the shipped adapter deciding, end to end: permission review on real agent commands, context compaction over a live session, browser-action scoring, and the official System One HTTP schema, with the probabilities on screen. Nothing in the clip is scripted; it is one run of the project's demo script, recorded automatically.

This is a 0.8B model we fine-tuned to make one narrow kind of decision very cheaply. It reads a state, a question, and a list of options, then returns a probability for each option. It never writes text. One forward pass, one token, and you have a distribution your code can branch on.

We built it for our own product first. An educational chatbot needed to decide, before spending a full LLM call, whether a student message was on topic, whether a tool call was safe to run, and whether an old tool result still had to be kept verbatim in the context. Those are small bounded judgments, and paying a 70B model to make them was absurd. It runs today on an office GPU at 80 to 150 ms per decision, and on a laptop with llama.cpp.

Files

The merged v19c files were converted with --no-mtp (the MTP head is useless for a one-token readout and the default conversion writes a broken block count); fidelity was measured row-by-row on the 222-question clean extended probe: the Q8_0 merge agrees with the base-GGUF-plus---lora path on 222 of 222 answers (89.2% both ways). Everything in this table shipped over the hf-mirror.com endpoint.

Fidelity was measured on our 180 question product set: the merged models agree with the base-GGUF-plus---lora path on 179 of 180 (f16) and 178 of 180 (Q8_0) decisions, at 94.4% and 93.9% accuracy. Quantisation costs one or two answers in 180, nothing more.

One thing to know about that 180: 29% of its rows appear verbatim in the training mix (38 of its 100 gate rows, 15 of its 30 permission rows; its email and compaction rows are clean). The agreement figures above are unaffected, since they compare two ways of serving the same weights, but the accuracy on that set is partly a memory score. The probes we trust for generalisation are the extended email and compaction sets, which do not overlap training at all (88.2% and 92.9% for the released adapter), and the external benchmarks in the table below.

release/fetch_model.sh in the project downloads the base GGUF plus the adapter GGUF and checks both hashes, so nothing here needs a full model to be repackaged.

What it does well, and what it does not

We ran the whole thing against the official Jev 1.13 API, against three open competitors, and against the base model without any fine-tuning. The short version is in the chart below.

Left two bars: our own task distribution, the one the model was trained for. Right two: held-out general benchmarks, where the base model is as good as our fine-tune.

On the distribution it was built for, fine-tuning is worth +67 points over the untouched base model (27.2% to 94.4% on 180 questions). On held-out general benchmarks it is worth nothing: the base model scores 50.6% on jev-bench held-out configs and our fine-tune scores 49.3%, with overlapping confidence intervals. The general axes belong to the backbone, and at 0.8B the backbone sits near 50%. The official API reaches 75.0% there. If general decision accuracy is what you need, use a larger model; the 4B prompt-based baselines land at 74 to 79% on JevBench.

We would rather write this down than let you find out after deploying.

Each point is a version we trained, scored on 231 public tasks (left) and 960 jev-bench rows (right). The solid line counts every source. The dashed line counts only sources the model never trained on. The two lines tell different stories about the same nine runs.

That gap is the most useful thing we learned. We spent two days improving a number that turned out to be mostly memorisation of the sources we trained on. After rebuilding the evaluation to hold out sources rather than records, the honest picture is: v1 to v11 gained 5 points on jev-bench held-out configs (44.3% to 49.3%) and 2.9 points on held-out JevBench families, where the confidence interval is 9 points wide. The rest was in-distribution.

Product family readout

The work we care about happens in four families: routing a message, judging a tool call, keeping or dropping context, and sorting an email. Each is measured on its own held-out questions. We extended the email and compaction sets past 100 questions each after discovering that 30 questions gave us a confidence interval 23 points wide.

Permissions and gate on the left, email and compaction on the right, both above 100 questions. Email is the hardest family, and the one the v14 data work targeted.

The four families are not equally hard, and that spread matters more than the average. On the extended probes gate and permissions sit at 97 to 100% and email and compaction at 74 to 79%. Email moved least across our first twelve training runs, which is what the v14 data work targeted; the released adapter now reaches 93.3% on the product email probe and 88.2% on the harder extended one.

One honest note about versions, and it cuts both ways. The released adapter is v14_s0, which beat the previous release (v5) on every product instrument in a ten-battery run on identical instruments: the frozen 180 question set 94.4% against 93.9% (one flipped answer against four), the 222 extended email/compaction probes 88.2% and 92.9% against 69.1% and 75.9%, permission review on real commands 87.2% against 58.1%, gates unchanged at 100%. Most of that gain turned out to be a fix rather than more knowledge: v5 answered permission questions at 80% in the option order it was trained on and 46.7% with the options reversed, and per-row option shuffling in v14_s0 removed that swing. The same shuffling is why v14_s0 handles the three way NLI format at 89.3% where v5 managed 41.3%.

The part that does not cut our way: that training recipe is not reliable. Four seeds of it produced two strong adapters, one that collapsed halfway and one that collapsed to the base model, and the training loss ranks the outcome almost perfectly, so any future retrain has to gate on it. v14_s0 is shipped as a validated artefact, not as a reproducible procedure. On the general reasoning axis nothing here beats the base model's ceiling either: held-out sources score the same before and after fine-tuning (about 50%), and JevBench lands at 57.6% for v14_s0 against 61.9% for v5, both far behind the official System One implementation. Every number in the tables above says which version it came from.

Injection hardening, and a same-base comparison (2026-09-27/28)

The gate's input is tool output and cannot be trusted. A forged [OPTIONS] block appended to the state flips 48 of 86 permission answers on the released adapter (87.2% to 39.5%); text-level sanitisation does not recover it (we tried escaping markers, neutralising option rows, removing brackets: all fail). Training with marked decoy blocks does: across three seeds the hardened adapters sit at 0.0 / +1.2 / 0.0 pp gap on the forged probe (0 to 2 answers flipped per 86), while unmarked-decoy seeds spread from −8.1 to −51.2 pp. On clean instruments the hardened line matches the released one (81.5–88.7% vs 90.5%, overlapping intervals), and its calibration on the product set is the best recorded in this project: ECE 0.0340 on 180 questions, on the hardened line (the v5 line reads 0.0427), against 0.0523 for v14_s0 (official-Jev reference 0.0313, different task). The hardened recipe also answers the v14 reproducibility complaint: all three seeds passed the loss gate and stayed near zero gap.

We also evaluated Intern-Decision-0.8B, a decision model fine-tuned from the same base (Qwen3.5-0.8B) on the same three primitives, under our two-pass option-swap protocol on our instruments: 52.3% on the 222 extended email and compaction probes (ours, v19 three seeds: 83.3–89.2%), 57.0% on real permission commands (ours 83.7–93.0%), 62.8% on the 180-question set (ours 93.9–94.4%), and ECE 0.335–0.467 (ours 0.03–0.11; banking-77 intent classification is at chance for both of us). Same base, same interface, same instruments: the domain-specific training data carries the line, not scale or interface. Their card reports 79.4 average on their own benchmark suite.

Prompt-cache-aware rendering (sysone2, confirmed across three seeds)

The rendering contract was re-derived so the state comes first ([STATE][QUESTION][OPTIONS]). On the old order the two option-swap passes share no KV prefix (0 of 2270 tokens hit at a 2270-token prompt); on the new order they hit 1754 (77%). A decision costs 39% less prefill; a ten-question batch costs 76% less. Changing the rendering changes the contract, so the mix was retrained under the new order (v18m, 40,290 rows: banking-77 removed after it measured at chance on the clean instrument, a 2,400-row parameter-level tool-permission family added). Three seeds passed the full battery at 86.9 / 86.0 / 84.2 with forged-probe gaps of +0.0 / −5.8 / +0.0 — overlapping the pre-switch line, so the rendering switch is confirmed. Old numbers remain valid for the old contract and are archived as such.

Since then (2026-09-29 → 10-07): a five-format RAG family, a calibration decomposition, and a 34-artifact baseline

Five-format RAG family (v19 line, three-seed verdict). A 4,800-row family teaching RAG answerability ("can this passage answer this query") now ships in five input formats: plain 40% / markdown 25% / raw HTML 20% / XML shell 7.5% / JSON shell 7.5%, with tag-hidden instruction decoys on raw rows. The format drift is measured, not assumed: markdown shifts p(yes) by +0.13 overall, raw HTML adds ±0.2 per-pair noise, structural shells are harmless — so cross-format coverage has to be trained, not prompted. Across three seeds (mix_v19) the extended probes read 87.8 / 83.3 / 89.2 (89.2 is the best of the entire 34-artifact lineage), the forged-probe gap is +1.2 on all three seeds with zero variance, and held-out family accuracy sits at 85.0 / 81.7 / 81.7 against 56.7 for a no-family control: learnable in distribution, at no cost to anything else. Cross-corpus transfer (DDnim, 36 questions) did not pass the pre-registered bar (gate unmet on 3 of 3 seeds, gold@1 {0, 7, 8}); that moves to a corpus-diversity follow-up.

Calibration decomposition (paired, same base). On identical instruments and the same readout, the base model sits at ECE 0.40–0.47 in domain while the fine-tuned line sits at 0.03–0.11, matching the 10×-larger official engine (0.031). On a held-out RAG probe the base model answers 52.5% (chance) with 0% gate coverage: it does not know what it does not know. Calibration is taught by fine-tuning, not carried by pretraining — and the out-of-distribution axis is where capacity matters (our 4B base: OOD ECE 0.050; our 0.8B: 0.373).

A 34-artifact baseline. The v18-era backfill re-measured 31 artifacts (base plus everything through v18) on one stack with the six-instrument battery (~9 minutes per artifact); the three v19 seeds ran the same battery on landing, making 34 artifacts in the comparable set. The historical numbers reproduce bit-for-bit. The out-of-distribution finding is lineage-wide: healthy artifacts pass the RAG answerability gate on 32–36 of 36 questions (i.e. gate everything), and only collapsed or failed arms ever reject — v19c is the first artifact trained to change that.

Latency, and how the alternatives compare

One protocol, p50 per decision. The official API number includes the network round trip; the local competitors were measured on their own recommended hardware.

Measured by us, each engine using its own prompt contract:

We are ahead where we trained and behind where we did not. For a model that has to fit in 812 MB and answer in under a tenth of a second, that trade seems right.

Reliability

149 of 180 questions land in the top confidence bin, and they are right 96% of the time. ECE 0.0427 (v5 line; the hardened line reads 0.0340, see the injection section). Confidence follows the official Jev formula, c = (pmax - 1/K) / (1 - 1/K), computed from the option letter distribution.

Confidence here is a statistic of the distribution, not a claim that the answer is correct. Use it to route: act above a threshold, escalate below. Out of domain it is over-confident, and so is the official API. On the three jev-bench configs where human annotators disagree with each other (ChaosNLI, hate speech vote shares, Civil Comments), no model we measured tracks human uncertainty, the official one included. We report total variation distance to the human distribution rather than pretending that axis is solved.

Quick start

Pair the adapter with the base model, which is the smallest download:

llama-server -m Qwen3.5-0.8B-Q8_0.gguf --lora qjev35-lora-v14_s0.gguf \
  --port 8080 -c 4096 --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

python qjev_client.py --url http://localhost:8080 \
  --state "The build failed: missing pyarrow in the test venv." \
  --question "What should the agent do next?" \
  --options "install the dependency" "retry the build" "ask the user"

Or run a merged file on its own:

llama-server -m qjev35-0.8b-q8_0.gguf --port 8080 -c 4096 --jinja \
  --chat-template-kwargs '{"enable_thinking": false}'

In LM Studio: download either merged GGUF, load it, start the local server from the Developer tab, and point qjev_client.py at http://localhost:1234 with the model name LM Studio shows.

qjev_client.py uses nothing but the standard library. It asks for one token with top_logprobs from /v1/chat/completions and renormalises the option letters:

from qjev_client import decide, decide_bool, decide_score

decide("http://localhost:8080",
       "Disk usage hit 96% on the build host.",
       "Should the agent stop the pipeline?",
       ["stop and clean up", "continue", "escalate to a human"])
# [('stop and clean up', 0.91), ('escalate to a human', 0.07), ('continue', 0.02)]

decide_bool("http://localhost:8080", evidence, "Does the passage entail the claim?")  # P(true)
decide_score("http://localhost:8080", ticket, "How urgent is this?", levels)         # distribution

Option order does not matter. The client evaluates the order you gave and the reverse order, then averages the two distributions. That trick is worth 2 to 4 accuracy points at this model size, and we have used it since the first version.

Prompt format

Set the chat template with thinking disabled. The model was trained on this shape:

system: You are a judgment engine. Read the question and the options, then decide which single
        option best matches the state. Answer with EXACTLY ONE letter and nothing else.

user:   [QUESTION]
        {question}

        [OPTIONS]
        A. {option 1}
        B. {option 2}

        [STATE]
        {state}

The next token is the option letter. Its probability, renormalised over the letters you declared, is your distribution. For yes/no questions pass two options (false, true). For ordered levels, list them in order. The released GGUF ships a pass-through chat template, so a chat endpoint hands this text to the model unchanged.

Read the letters from log probs. Sampling text and parsing it throws away the whole point.

Scope

Good for bounded decisions where you know the options when you call the model: tool gating, permission allow/ask/deny, prompt injection and content checks, routing, context compaction, email triage, picking an entity, browser actions, and four ordered score dimensions.

Not for open-ended reasoning, general knowledge, mathematics, multi-hop planning, long document question answering, or judging free-form assistant output. The held-out numbers above are the reason, and they are properties of the 0.8B backbone.

Option cap is 26, using positional letters A to Z. The official Jev allows 255. That is a design limit rather than an oversight. Two separate reasons stop us running 77 way (Banking77) or 151 way (clinc150): the letter readout caps at 26 options, and, measured after this card was first written, the adapter answers a 25-option Banking77 question drawn from the full label space at 3.9% (154 rows, chance is 4.0%). Our earlier Banking77 numbers came from rows whose distractors were a small sampled subset, and most of those rows sat in the training mix, so they measured memory tasks. We report it as a limitation instead of working around it with a hack.

Training data and license

Seventeen families, all synthetic or converted from public datasets: gate, permissions, compaction, email, entity, browser actions, four score dimensions, NLI (SciTail, MNLI, ANLI), CLINC, Banking77, MASSIVE Chinese, ToolACE, WANLI, CMNLI, tldr-pages, and nl2bash. Per-source licences are listed in SOURCES.md in the dataset repository. ANLI (cc-by-nc-4.0) is used for internal training only and is stripped from every published export. Weights are Apache-2.0, following Qwen/Qwen3.5-0.8B.

How we evaluate

Five rules, written down after we broke them:

1. Hold out sources, not records.
2. Treat any difference smaller than its confidence interval as no evidence.
3. Fit on validation, report on test.
4. Require at least three training seeds before claiming one version beats another.
5. Keep domain numbers, held-out numbers, and competitor numbers in separate columns.

They are the reason the tables above are less flattering than our first release notes, and the reason we trust them more.

Known limitations

• Held-out general accuracy sits near 50%, which is where the 0.8B backbone sits.
• Five and six level ordinal scales that we did not train on score near chance, 25 to 27%.
• Confidence is over-confident out of domain, on our model and on the official API alike.
• The model leans on canonical option order more than one trained on shuffled orders would. Keep permutation averaging on, and if you train on this data yourself, shuffle the options per row. We found a fixed option order baked into 1,953 rows of our own training data; fixing it moved the position uniformity statistic from z = +6.6 to z = -0.2.

Citation

@misc{qjev35,
  title  = {QJev 3.5-0.8B: a small decision engine},
  author = {Lee, Jerry and contributors},
  year   = {2026},
  url    = {https://huggingface.co/ZY-Lee4/QJev3.5-0.8B-GGUF}
}

Independent work, written from scratch against the publicly described System One pattern. Not affiliated with, endorsed by, or connected to TypeSafe AI.