JevK5-2B v0.2: the small JevK5
JevK5-2B is the 2-billion-parameter version of JevK5,
an Apache-2.0 open alternative to TypeSafe's Jev for typed decisions. It reads a document and a
yes/no (noul), choice, or score question and returns a probability for every option in one
forward pass, with nothing generated. It is not affiliated with TypeSafe AI.
It was trained on exactly the same data, the same way, as JevK5 v0.2 (4B), on the smaller Qwen3.5-2B. It keeps most of JevK5's accuracy at half the memory: good at reading a document and deciding, weaker on arithmetic, multi-step reasoning, and general knowledge.
- Base: Qwen3.5-2B, with a LoRA (rank 16, attention and linear-attention projections) merged into the weights
- Readout: a softmax over the answer letters' next-token logits, divided by one calibration
temperature (
jevk5_config.json, T = 1.42) - Memory: 3.5 GB of bf16 weights, about 3.8 GB at peak on a 4,000-token document; a 2.0 GB GGUF (Q8_0) is in JevK5-GGUF
- Speed: about 9 ms per decision on an H100 with the JevK5 runtime
- License: Apache-2.0
Results
Held-out teacher questions. 362 questions from three business domains that training never saw (residential leases, public-sector permits, manufacturing QC). This is the set used to choose between models and to fit the temperature.
| JevK5-2B | JevK5 (4B) | |
|---|---|---|
| All held-out teacher questions | 0.751 | 0.804 |
| Extraction | 0.91 | 0.94 |
| Routing | 0.97 | 0.89 |
| Trade-offs | 0.88 | 0.88 |
| Judging answers | 0.70 | 0.73 |
| Multi-step lookups | 0.67 | 0.76 |
| Ambiguous questions | 0.59 | 0.76 |
| Dates and numbers | 0.42 | 0.58 |
Each question family has only 21-42 items, so the per-family rows are rough.
JevBench's 231 public decisions, through JevBench's own runner and scoring. Reported only; never used to choose anything. These are our runs, not official results: JevK5-2B has not been submitted to JevBench.
| Split | n | JevK5-2B | JevK5 (4B) |
|---|---|---|---|
| easy | 48 | 1.000 | 1.000 |
| original (standard) | 72 | 0.806 | 0.958 |
| hard (public half) | 111 | 0.604 | 0.739 |
| hard-tier calibration error (ECE) | 0.071 | 0.066 | |
| distance to gold on probability items (TVD) | 10 | 0.241 | 0.196 |
How it was trained
The same data and recipe as JevK5 v0.2. Qwen3.6-27B, with thinking on, wrote realistic documents with hard typed questions across 17 business domains and answered every question twice, independently; a question was kept only when both answers matched the intended one. Training used 3,272 of those questions plus as many human-labelled items from MMLU-Pro, WANLI, MultiNLI, BoolQ, banking77, ARC and CommonsenseQA: cross-entropy on the option-letter logits, 2 epochs, learning rate 3e-5. The temperature was fitted on the held-out teacher questions above.
We also tried, at 2B, distilling JevK5's own option probabilities, three option orders per question, 3,400 extra replay questions, and a higher learning rate with a larger adapter. None improved the held-out teacher questions, so this is the plain recipe.
No JevBench item, public or held out, and no output of Jev was used for training, tuning or
selection. Two corrections, shared with JevK5, are described in CHANGELOG.md in the
repository:
- Our hand-written calibration set echoed a public JevBench instruction and one public item's rule wording. Both are rewritten.
- The 940 MMLU-Pro items in the training data came from MMLU-Pro's test split, since it has no training split. JevK5 (4B), trained on the same items, scores 5.5 points higher on them than on MMLU-Pro items it never saw; the 2B has not been measured. The next version uses no test split.
Known weak spots
- Dates, amounts and multi-step lookups: the largest gap to the 4B model.
- The standard tier drops to 0.806 from the 4B's 0.958.
- Not measured on JevBench's sealed decisions, where JevK5 (4B) drops to 33.1% and every one-pass model struggles. Expect the same or worse here.
- English only. Inputs over 16,384 tokens are refused, not cut.
Use
With the JevK5 runtime on an NVIDIA GPU:
pip install "jevk5[fast] @ git+https://github.com/allebee/jevk5@v0.2.0"
from jevk5 import JevK5
model = JevK5("alibiserikbay/JevK5-2B")
model.decide(
"I was billed twice for order #4411. Please refund the duplicate charge today.",
{"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "Payments and refunds", "tech": "Bugs", "sales": "New purchases"}},
)
# {'type': 'choice', 'choice': 'billing', 'confidence': 0.99, ...}
On other GPUs, on a Mac, or on a CPU, use the GGUF build with llama.cpp: see JevK5-GGUF.
Credits
Qwen3.5-2B and Qwen3.6-27B by the Qwen team (Apache-2.0). The one-pass readout and prompt come from SemIf by TheoLeeCJ (MIT). Evaluated with JevBench (github.com/fstandhartinger/jevbench, MIT). Not affiliated with TypeSafe AI or Jev.