Kahn1-Qwen3.5-4B
Kahn1 is a calibrated "System 1" decision engine: it reads a document (the state) and
answers typed questions about it in one forward pass, from the logits of the option tokens,
without generating or parsing text. This is the 4B model, a LoRA fine-tune of
Qwen/Qwen3.5-4B merged into a standalone text-only checkpoint (Qwen3_5ForCausalLM).
Both recorded with this model served locally with vLLM (k = 3 for Snake). Try them on kahn1.com, where the in-browser versions run a 4-bit GGUF of the 3B.
- No schema or typing errors: every answer is one of the options you gave, as a typed value.
- Three typed primitives: Choice (up to 26 options + fallback), Score (ordinal levels, continuous expectation), Noul (does the document support a statement).
- Calibrated probabilities (temperature scaling,
calibration.json). - English and French.
Website, playground and benchmarks: https://kahn1.com · Engine, server, training and evaluation code: https://github.com/Okura66/kahn1
At a glance
| Kahn1 4B | |
|---|---|
| What it does | Typed answers (Choice, Score, Noul) about a text, each with its distribution and a calibrated confidence |
| Base | Qwen/Qwen3.5-4B + LoRA (r = 16), native chat template, thinking off |
| Held-out, 14,663 items, same options for every system | 72.4 % (JEV 1.13.0: 73.2 %) |
| JevBench, 231 public items | 87.4 %, on par with Jev's 86.6 % (exact McNemar p = 0.84) |
| Calibration (ECE, held-out) | 0.072 overall, 0.015 on Choice |
| Latency, k = 3, one RTX 5070 Ti | p50 88.7 ms, p95 279.3 ms |
| Languages | English, French |
| Licence | Apache 2.0 weights, MIT code |
Quickstart
git clone https://github.com/Okura66/kahn1 && cd kahn1
uv venv --python 3.11 && uv pip install -e ".[gpu]"
SYSONE_MODEL=Okura66/Kahn1-Qwen3.5-4B uv run sysone serve --port 8000
curl -X POST http://127.0.0.1:8000/v1/evaluate \
-H "Content-Type: application/json" \
-d '{
"state": "Hello, I cannot log in to my account since this morning.",
"questions": [
{"kind": "choice", "key": "category", "prompt": "Support ticket category?",
"options": ["bug", "billing", "account"], "allow_other": true},
{"kind": "score", "key": "urgency", "prompt": "Level of urgency",
"levels": ["low", "medium", "critical"]},
{"kind": "noul", "key": "legal_threat",
"statement": "The customer makes an explicit legal threat"}
]
}'
The engine detects the Qwen3 prompt layout from the model name or its config.json.
calibration.json holds the temperatures it applies (SYSONE_CALIBRATION). A LoRA adapter of the
same weights is published at
Okura66/Kahn1-Qwen3.5-4B-LoRA. The 3B
size of the same model is at Okura66/Kahn1-Qwen2.5-3B
(Qwen Research License).
Benchmarks
Every comparison is paired: the same items, the same prompts, the same labels. Kahn1: debiasing over k = 3 option orders and temperature calibration.
JevBench, 231 public items
| Tier | Kahn1 4B | Kahn1 3B | JevK5 v0.2 | Jev 1.13.0 |
|---|---|---|---|---|
| Easy (48) | 100 % | 100 % | 100 % | 100 % |
| Standard (72) | 97.2 % | 84.7 % | 95.8 % | 98.6 % |
| Hard (111) | 75.7 % | 42.3 % | 73.9 % | 73.0 % |
| All | 87.4 % | 67.5 % | 86.1 % | 86.6 % |
These figures come from three different runners on the same items: ours for Kahn1 (ECE 0.050), the outcomes JevBench publishes for Jev, and JevK5's authors' own published run for JevK5. JevBench's own run of JevK5 v0.2 scores 85.3 % on the same 231 items (v1.4 aggregates). Item by item, Kahn1 4B is on par with both: 13 items only Kahn1 4B gets right and 10 only JevK5 (exact McNemar p = 0.68); 13 only Kahn1 4B and 11 only Jev (p = 0.84). Neither gap is significant.
Held-out, 14,663 items (sources never seen in training)
Choice is asked over the same 8 options for every system (the gold one and 7 distractors), and JEV's Noul is asked whether the text supports the statement, as Kahn1's is.
| Kahn1 4B | Kahn1 3B | JEV 1.13.0 | |
|---|---|---|---|
| Choice (banking77, MASSIVE), 6,050 | 92.3 % | 90.9 % | 94.5 % |
| Score (SST-5, app reviews, 5 levels), 6,210 | 51.9 % | 51.4 % | 52.0 % |
| Noul (RTE, SciTail), 2,403 | 75.5 % | 67.0 % | 74.4 % |
| All | 72.4 % | 70.3 % | 73.2 % |
| ECE (15 bins, lower is better) | 0.072 | 0.082 | 0.113 |
Over every intent (77 / 60 options, same 1,184 items): Kahn1 4B 71.6 %, Kahn1 3B 68.4 %, JEV 79.1 %.
| Source | Kahn1 4B | JEV 1.13.0 |
|---|---|---|
| banking77 (8 options) | 91.8 % | 94.8 % |
| MASSIVE (8 options) | 92.9 % | 94.1 % |
| SST-5 | 55.0 % exact · 94.5 % off-by-one · Spearman ρ 0.833 | 57.7 % exact |
| App reviews (5 stars) | 50.2 % exact · 84.4 % off-by-one · Spearman ρ 0.783 | 48.9 % exact |
| RTE | 85.9 % | 89.9 % |
| SciTail | 74.1 % | 72.4 % |
Hard decision dev split, 317 items (k = 1)
Questions written by Claude Opus and checked by two blind solvers; never trained on, but used to choose the checkpoint: read it as a dev score, not a benchmark.
| Qwen3.5-4B base | Kahn1 4B | |
|---|---|---|
| Choice (146) | 46.6 % | 67.1 % |
| Score (75) | 42.7 % | 57.3 % |
| Noul (96) | 57.3 % | 76.0 % |
| English (233) | 48.9 % | 67.0 % |
| French (84) | 48.8 % | 69.0 % |
| All, balanced over primitives | 48.8 % | 66.8 % |
Where it is weak
- JEV is ahead on the held-out set overall (73.2 % against 72.4 %), on Choice (94.5 % against 92.3 %, and 79.1 % against 71.6 % over every intent); the two are level on Score (52.0 % against 51.9 %).
- Date, duration and amount computations: a single forward pass does not reason step by step (5 of 15 on JevBench's hard temporal items).
- Score is its weakest primitive (51.9 % exact on 5 levels, 84 to 95 % within one level).
- Calibration was fitted on the training distribution: recalibrate on your domain
(
sysone calibrate) before trusting a threshold.
Latency: p50 88.7 ms, p95 279.3 ms, 8.5 queries/s at k = 3 on one RTX 5070 Ti with vLLM, measured during the held-out run (3B: 36.4 ms).
Training
| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen3.5-4B, text only, bfloat16 |
| Prompt | native chat template, thinking off, answer read right after </think> |
| Method | LoRA r = 16, α = 32, on q/k/v/o of the attention layers and in_proj_qkv/z/b/a, out_proj of the linear-attention layers |
| Loss | cross-entropy restricted to the candidate tokens (as inference reads them) + 1.0 × squared EMD on ordinal Score questions |
| Optimizer | lr 3e-5, cosine schedule, effective batch 32, two epochs (2,010 steps) |
| Data | 32,170 rows: public NLI, topic, intent and sentiment sources (ShARC, WANLI and DocNLI whole), QuALITY, BoolQ, CLINC150; replay of ARC (1,000), CommonsenseQA (1,000) and the 56 MMLU-Pro validation questions (never MMLU-Pro test); teacher questions, 20.7 % of the rows |
| Teacher questions | 2,836 distinct questions written by Claude Opus on long, realistic business documents, English and French (402 × 3, 600 × 3, 1,834 × 2). The newest 1,834 were deduplicated against the earlier questions and JevBench, and a random sample of 150 was re-solved blind by a second model (Claude Sonnet), which disagreed on 0.7 % |
| Augmentation | option permutation, option dropping with a fallback, bilingual templates for short tasks |
| Checkpoint | final step, chosen on dev accuracy (two hard dev splits and a mixed one), never on a benchmark |
License
Apache 2.0 (inherited from Qwen3.5). Engine code: MIT. Training sources include ShARC and BoolQ (CC BY-SA 3.0), QuALITY and WANLI (CC BY 4.0), CLINC150 (CC BY 3.0), ARC (CC BY-SA 4.0), CommonsenseQA and MMLU-Pro (MIT) and DocNLI; the teacher questions were written with Claude.