Kapteeni v1 (kapteeni-v1-meticulous) — a Jev-compatible System One decision model
Send a state plus typed questions; get back calibrated probability
distributions your code can branch on. No text generation. Kapteeni
implements the TypeSafe System One decision-model
interface (the wire format of Jev, POST /v1/systemone).
This variant: kapteeni-v1-meticulous — conservative confidence. The safe default for unknown or messy traffic.
Kapteeni v1 ships as two variants of the same architecture. The other, kapteeni-v1-intuit, decides more accurately on well-formed numeric, temporal, and multi-step policy traffic but is less careful about its confidence there.
Quickstart
pip install torch transformers safetensors huggingface_hub # plus the kapteeni package below
python -m kapteeni.serve --dist . --port 8000 # run from this snapshot
curl localhost:8000/v1/systemone -d '{
"state": "Help! My payouts have been failing for 3 days.",
"model": "jev-latest",
"questions": {
"is_urgent": {"type": "noul", "instructions": "Does this convey urgency?"},
"department": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": "Payments, invoicing, refunds",
"technical": "Bugs, outages, integrations",
"sales": "Pricing, upgrades, new accounts"}},
"frustration": {"type": "score", "instructions": "How frustrated is the customer?",
"criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry"]}
}}'
Or in-process:
from kapteeni.model import SystemOneModel
m = SystemOneModel.from_dist("<snapshot dir>")
answers, usage = m.evaluate(state, questions)
The kapteeni/ Python package is included in this snapshot; the full
repository (data pipeline, training, tests, benchmark history and
pre-registrations) lives at https://github.com/TriusAI/Kapteeni
(Apache-2.0).
Question types (the Jev contract)
| type | answer | readout |
|---|---|---|
noul | {type, noul} — P(yes) in [0,1] | absolute: sigmoid head on the final hidden state; not complement-consistent (P(A) + P(not-A) may differ from 1, matching the reference) |
choice | {type, choice, probabilities, confidence} | relative: per-option passes through the backbone, shared head, group softmax; probabilities sum to exactly 1 |
score | {type, score, legend, probabilities, confidence} | independent levels; score is the probability-weighted expectation and can land between levels |
Serving adds a verbalizer blend: the backbone's own next-token yes/no
logits are geometrically blended with the trained head (weights in
kapteeni-config.json), which measurably improves out-of-domain robustness
and calibration. --readout head or --readout verb select pure variants.
Benchmark (JevBench v1.4 public half, self-reported)
231 public decisions, scored with the benchmark's own code, end-to-end through the server. Numbers only — no placement claims; see caveats.
| JevBench-style score | Intelligence | top-label ECE | public accuracy |
|---|---|---|---|
| 65.71 | 60.3 | 0.0496 (Calibration 90.1) | 0.710 (easy 1.000 / standard 0.889 / hard 0.469) |
Axes common to both variants: Speed 81.0 (p50 0.17 s, p95 1.2 s on an AMD Strix Halo iGPU, x2 self-hosted adjustment) · Cost: 597 input tokens/decision at the hosted list price — verified 2026-09-26: $0.03/M (Novita, qwen3-4b-fp8) = $0.018/1k decisions; nearest official Alibaba tier (qwen-turbo, $0.05/M) = $0.030/1k; an earlier $0.14/M assumption ($0.083/1k) is kept as the pessimistic bound. Under the benchmark's published Jev-class definition (cost <= $0.080/1k decisions, adjusted median latency <= 1.30 s), both variants qualify at verified prices (adjusted p50 0.34 s). Composite = harmonic mean; the Intelligence<50 gate does not apply.
Caveats, stated plainly: self-reported public half (judge and sealed items are private; not an official rank); all serving constants pre-registered on held-out validation (no benchmark selection); Intelligence is renormalized without the sealed judge tier; Calibration shown is the ECE half only. Differences of a few points are within single-seed pipeline noise (~±2-3 composite points, measured).
How it was trained
Qwen3-4B-Instruct-2507 (frozen for the data pipeline, then) LoRA r=32 on all attention and MLP projections for one epoch over an 8.7M-token decision mix: BoolQ + FEVER (teacher-soft-labeled with k=5 sampled agreement), Banking77, CLINC150, GoEmotions, HelpSteer2, and 3.6k synthetic temporal/numeric/policy items with ground truth by construction. Readout heads (2-layer MLPs on the final hidden state) trained with proper scoring rules only. Per-head temperature scaling; blend constants fit on held-out mixed-domain validation. Out-of-domain gate: MNLI excluded from training, held 0.88 -> 0.893 through all 1,205 steps.
Training data provenance
| data | license |
|---|---|
| Qwen3-4B-Instruct-2507 (base) | Apache-2.0 |
| GoEmotions | Apache-2.0 |
| Banking77, HelpSteer2 | CC BY 4.0 |
| BoolQ | CC BY 3.0 |
| MultiNLI | CC BY-SA 4.0 |
| FEVER (underlying) | CC BY-SA 3.0 |
| CLINC150 | research use (no explicit license) |
| synthetic items | Apache-2.0 (this project) |
Weights: CC BY-SA 4.0 (the ShareAlike terms of MultiNLI/FEVER carry the
recommendation; see WEIGHTS-LICENSE.md for the full attribution
obligations). Code: Apache-2.0 (LICENSE).
Limitations
- English-primary; long-policy and multi-hop reasoning remain weak (hard-tier accuracy 0.469); temporal/numeric judgment is unreliable on this variant — prefer -intuit for such traffic.
- Probabilities are calibrated in aggregate; individual answers are not guaranteed correct — branch on confidence where it matters.
- Not affiliated with or endorsed by TypeSafe AI; "Jev" is their model and trademark; this is an independent implementation of the documented interface, evaluated on the public JevBench items.