ballot-jev-0.5b

A small typed-decision model whose answer does not change when you reorder the options.

Decision models take some context and return a structured answer from a fixed set of candidates — route this ticket, is this claim supported, rate this response 1–5. Almost all of them share one quiet bug: feed the same options in a different order and the answer changes. ballot-jev structurally cannot. The state and question are encoded once into a KV cache, and every option is then decoded from its own private clone of that cache. No option is ever an input to another option's computation, so permuting the option list permutes the output distribution and does nothing else. Verified on the trained weights: max |p(forward) − p(reversed)| = 4.5e-8, float32 rounding noise.

This is v2. The v1 checkpoint is preserved at the v1 tag; see What changed in v2.

At a glance

It is a 0.5B-class model: small enough to sit next to your application on an ordinary CPU box, with no GPU, no server, and no per-call API bill.

Latency scales with the number of options

Each option is decoded from its own branch off the shared cache, so cost grows roughly linearly with K. Plan for the K you actually use — a wide roster (20+ candidates) is over a second per decision on CPU, not a fraction of one.

Measured on an Apple Silicon CPU, 4 threads, single request, otherwise idle. Roughly +46 ms per additional option.

The v1 card quoted 0.19 / 0.30 / 1.44 s for the same shapes. The architecture is unchanged, so this is not a speedup — those were taken while other work was running; these were taken on a verified-idle machine.

What changed in v2

Same architecture, same size, same invariance guarantee. Three training changes, each chosen by measurement rather than by guess, and evaluated on the identical 750-record validation set v1 was scored on:

Eight of ten sources up; the two that dipped moved by one item each (75 per source).

1. Brier weight 1.0 → 0.25. v1's loss carried a Brier term at full weight. An ablation on the encoder arm (three weights, identical data) showed it barely moves the aggregate — but broken out per source, it wrecks natural-language inference: at weight 1.0 the model answered "neutral" on 62 of 75 MNLI items and never once predicted "entailment." Proper-scoring pressure that strong makes the safe middle label a local optimum. At 0.25 that collapse is gone — at a measured cost to calibration, covered next.

2. MNLI ×3 (300 → 900 training examples). The per-question gradient normalisation in the loss gives multi-question sources (typed-decisions carries five per record) proportionally more weight, which had diluted MNLI's share to roughly 5%. Tripling it restores the share it had in the earliest, strongest encoder runs.

3. Two epochs instead of one. v1's loss was still falling at the end of its single epoch; validation accuracy in v2 was still climbing at step 700 of 900.

What v2 traded away: calibration. Lowering the Brier weight bought accuracy and cost sharpness. On JevBench's hard tier, expected calibration error went from 0.054 (v1) to 0.114 (v2). v2 still leads every CPU-class peer on that axis (laya 62.5, openJev-verdict 51.3 on JevBench's calibration score; v2 77.3), but v1 was better at it, and its confidences were the more literal ones. If calibrated probabilities matter more to you than raw accuracy — e.g. you threshold aggressively and the tail goes to a human — load the v1 tag. If you want the model that is right more often, this is it.

What did not change, deliberately: banking77, sst5, boolq and pubmedqa stay excluded from training as benchmark-contamination risks.

Install

pip install torch transformers peft safetensors huggingface_hub

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("vigneshlabs/ballot-jev-0.5b")   # ~2 GB
sys.path.insert(0, path)

from inference import BallotJev
jev = BallotJev.from_pretrained(path, threads=4)     # CPU by default

The Qwen2.5-0.5B base is fetched from the Hub automatically the first time (another ~1 GB). To pin your own copy: BallotJev.from_pretrained(path, base_model="/your/Qwen2.5-0.5B"). For GPU: device="cuda". For the previous release: snapshot_download("vigneshlabs/ballot-jev-0.5b", revision="v1").

The three question types

Every output below is this checkpoint's real, unedited response to that exact call.

# 1. choice — one of K
jev.choice(
    state="Customer: 'Where is my package? Ordered last week, still nothing.'",
    question="Which intent does this express?",
    options=["track order", "cancel order", "change address", "billing question"],
)
# {'answer': 'track order', 'index': 0, 'confidence': 0.540,
#  'probabilities': {'track order': 0.540, 'cancel order': 0.035,
#                    'change address': 0.139, 'billing question': 0.287}}

# 2. noul — true/false under evidence
jev.noul(
    state="Policy: refunds need a receipt and purchase within 30 days. "
          "Customer bought 12 days ago but has no receipt.",
    proposition="Under the stated policy, is a refund permitted?",
)
# {'answer': False, 'probability_true': 0.271, 'confidence': 0.729}

# 3. score — ordinal rating
jev.score(
    state="Shipping took nine days, but the product itself is flawless.",
    question="How many stars does this review give?",
    levels=["1 star", "2 stars", "3 stars", "4 stars", "5 stars"],
)
# {'answer': '4 stars', 'index': 3, 'expected_level': 3.39, 'confidence': 0.470,
#  'probabilities': {'1 star': 0.003, '2 stars': 0.011, '3 stars': 0.046,
#                    '4 stars': 0.470, '5 stars': 0.470}}

That last one is worth reading closely. The model puts 0.470 on both 4 and 5 stars — a genuine tie, broken by the content-based tie-break rather than by position — and expected_level 3.39 says "good, not perfect", which is the honest read of a mixed review. v1 spread the same input almost flat across 1–5; v2 has learned where the mass belongs. For ordinal questions, expected_level is usually the number you want.

Every call returns the full distribution, not just a label. That is the point — see What v2 traded away above for how literal those probabilities are.

The invariance, if you want to check it yourself

opts = ["track order", "cancel order", "change address", "billing question"]
a = jev.choice(state, question, opts)["probabilities"]
b = jev.choice(state, question, list(reversed(opts)))["probabilities"]
print(max(abs(a[k] - b[k]) for k in a))   # 4.5e-08 locally, 1.8e-07 from a clean Hub
                                           # download -- float32 noise either way

Shuffle, reverse, rotate — the number stays at float-noise. The exact digits depend on which build of the Qwen base you load; the guarantee does not.

Using it for real work

Where it fits. A cheap, local, well-calibrated first pass on high-volume typed decisions, in front of something bigger. Seven measured examples ship in results/demo_examples_v6.json; the model's answer and confidence for each:

Ten scenarios were written; seven ship. The three cut were cut because the model got them wrong, not because they didn't fit — see Limitations.

The pattern that makes it pay for itself — route on confidence, not on the label:

r = jev.choice(state=ticket, question="Which team handles this?", options=TEAMS)

if r["confidence"] >= THRESHOLD:
    assign(r["answer"])            # no API call
else:
    escalate_to_llm(ticket)        # only the genuinely hard ones

THRESHOLD depends on K and must be tuned on your own data — probability mass spreads thinner over more options.

A real workload, at K = 20. Six wire-service headlines routed to a 20-client roster (results/demo_newsdesk_v6.json). Four of six went to the expected client. The two that didn't scored 0.501 and 0.353; the four that did scored 0.684–0.924. At a 0.6 gate, every auto-filed story was correct and both misses landed in the review queue. That threshold was chosen after seeing these six results, so treat it as an illustration of the pattern, not a tuned production value.

Do not use it for multi-hop reasoning over long policies, arithmetic or temporal deduction, anything safety-critical without a human, or as a general chat model. It has no generative head — it scores candidates, it does not write.

Evaluation

All numbers below were produced on this machine against real downloaded weights and each project's own inference code — never a reimplementation, never a figure copied from someone's README.

Head-to-head — identical held-out set

140 records, stratified 10 from each of 14 sources. Flip rate is measured the way kev's own evaluation defines it: reverse the option list, check whether the same option text still wins.

Read the accuracy column with care. ballot-jev trained on 10 of these 14 sources and kev on 6, so the aggregate partly reflects training overlap. Per source, with * marking sources in that model's training mix:

laya still wins mnli, clinc_oos and yelp; kev still wins banking77, boolq and mnli — the sources it trained on and v2 did not, plus NLI, which remains ballot-jev's weakest trained source even after the v2 fix. Most of v2's margin comes from ordinal and evidence tasks the others handle poorly (summeval, helpsteer2, vitaminc, typed-decisions). Ten records per source: one item is ten points, so treat individual cells as indicative and the 750-record table above as the reliable like-for-like.

The flip-rate column carries no such caveat. It is architectural, it is not a sampling artifact, and it held after training: 0 of 32 for both v1 and v2.

JevBench (public subset)

Run through JevBench's own harness and scoring code on its 231 MIT-licensed public tasks.

Public tiers, v1 → v2: easy 0.938 → 0.958, standard 0.444 → 0.569, hard 0.342 → 0.351. Hard-tier ECE 0.054 → 0.114.

* Intelligence is computed over the three public tiers only and renormalised, because the judge tier is entirely held out. It is reported for completeness and is not comparable to a leaderboard row; the peers' figures are their published four-tier numbers. Cost reuses the per-1000 price JevBench itself assigned to the same local size class, not a figure derived here.

Three limits on any JevBench number here: only 231 of 534 tasks are public, the entire judge tier is held out (146 items, 28% of the Intelligence axis), and speed was measured on Apple Silicon rather than the project's reference box. This is a public-subset measurement, not an official leaderboard score, and the Intelligence axis is reported but not claimed as comparable to a leaderboard row.

Reproducing all of it

Every number in this card traces to a file in results/ or a script in the training repo's bench/. The invariance gate is bench/verify_invariance.py; the head-to-head is bench/final_comparison.py on the shared record set bench/eval_records.jsonl; JevBench runs through its own CLI with the ballot_local adapter and is scored by bench/jevbench_score.py.

How it was built

Architecture. Qwen2.5-0.5B with LoRA (rank 16, α 32, on q/k/v/o_proj + gate/up/down_proj), plus a listwise set-attention head over the option branches. state + question is encoded once with use_cache=True; each option then runs a short forward pass over a .clone() of that cache. The clone is load-bearing in two directions — it keeps options isolated from each other, and because it is a clone rather than a detached copy, gradients from every branch still flow back into the shared prefix.

Training. Two epochs, cross-entropy stage, on CPU. Loss is generalized CE against the full gold distribution where a source ships one, plus a Brier term at weight 0.25, plus an ordinal term for score. Gradients are normalised per question. 3,600 training records over 10 public sources: agnews, mnli (×3), yelp, clinc_oos, civil_comments, typed-decisions, vitaminc, Aegis, HelpSteer2, summeval. emotion is held out entirely as a zero-shot source.

Limitations

• Ordinal judgement is weak, even though ordinal rating improved. Asked to grade a helpful LLM answer and a deliberately useless one, it rated both "5 — excellent"; asked to rank six very different news stories for newsworthiness, it returned 1.91–2.07 out of 4 for all of them. Star ratings with a clear scale work; "how important is this" does not.
• Content moderation in the "personal attack" framing is unreliable. On four probe posts (results/moderation_probe_v6.json) it got two wrong, one of them confidently. Its in-distribution toxicity accuracy is high (0.907 on val); that particular question is not something it does reliably.
• Weak at reasoning that needs inference chains — multi-hop, long-policy, temporal.
• fp32 only, 2.02 GB. No quantized build yet. English only.
• Scores candidates, does not generate. You must supply the option set.
• Cross-model accuracy comparisons are shaped by training-mix overlap — read the per-source table, not the headline.

Citation

@software{ballot_jev_0_5b_2026,
  title  = {ballot-jev-0.5b: order-invariant typed decisions},
  author = {Varadharajan, Vignesh},
  year   = {2026},
  url    = {https://huggingface.co/vigneshlabs/ballot-jev-0.5b}
}

Built on Qwen2.5-0.5B (Apache-2.0).