SelfJev-4B — merged weights

Structured decisions from text. A 4B backbone. Your own GPU.

Route a customer message. Check a claim. Review an AI response. SelfJev answers questions over the options you supply and returns probabilities through a Python SDK or HTTP API. Its default engine shares the document computation across questions and scores candidates directly, without generating an answer token by token.

Quickstart · Results · Public datasets · Model size · Architecture · Latency · Research · Source code

This repository contains the complete merged checkpoint (approximately 9.3 GB of weights), tokenizer and configuration. The SelfJev LoRA is already incorporated into the pinned Qwen3.5-4B weights: no separate base or adapter download is needed. The original checkpoint layout, including unmodified vision weights, is retained; SelfJev is trained and evaluated for text.

Compact adapter release · Merge provenance · Integrity checks

Results

Reference results for the source adapter through TreeServer. The same charts and evaluation evidence are shared with the adapter release. This merged artifact passed tensor-integrity checks and two synthetic GPU smoke cases; it has not been benchmarked separately through vLLM.

What these measure. Text decisions cover authored routing, policy, evidence and difficult language cases. AI response review covers verification, judging, scoring, guardrails and jailbreak detection. Public-source tasks cover 11 adapted datasets. Each score counts a question as correct only when the expected answer matches; for multiple selections, the entire set must match.

These are recorded results from the current TreeServer engine and matched Jev predictions, not scores from a newly merged vLLM release. The authored sets use AI-checked labels, not human ground truth. Both have informed research direction. The public-source set is a repeatedly reused development benchmark. A fresh independent evaluation remains necessary.

How does it compare with another open model?

On the same 1,991 text-decision questions, the recorded Eikos-4B run scored 92.8% using our task adapter and its letter-based readout. This is a comparison under this project's protocol, not a general ranking of open models. Eikos report

S1Bench (the 13 public subsets pinned by Nimble, 3,880 items, run through lev's harness on 2026-09-30): SelfJev-4B Vision 74.8 macro accuracy; lev's card reports 68.9 for lev and 76.1 for Jev through the same harness. The other shared benchmarks: Compared with open Jev-like models.

Compared with open Jev-like models

Measured with SelfJev-4B Vision, the default release, which continues this adapter; on text the two score within half a point of each other (Text Decisions 96.1 vs 95.7).

Measured 2026-09-30. Eight open "Jev-like" decision models from Hugging Face and SelfJev were each served with their card's recommended setup on the same GPU (one NVIDIA L40S) and asked exactly the requests TypeSafe's Jev receives. A request a model cannot answer counts as wrong. Jev is shown for reference.

SelfJev's test suites. Text Decisions (1,991 questions) and AI Response Review (946), published as selfjev-decision-bench, and photos:

• Every model gets the same Jev-shaped questions; select-all questions become one yes/no per option, which costs SelfJev 1.4–1.8 points against its native evaluation (96.1 and 92.5). Held-out images: the 4 image datasets of the SelfJev image test that none of these models trained on (772 questions). Times: each suite end to end, 4 requests in flight; SelfJev 0.4.2 was measured 2026-10-05 on the same GPU model, and the row for version 0.4.1 is the same weights on the engine before it. ² Refuses inputs over 8,192 tokens (2 % and 1 % of the questions), counted wrong.
• No other model trained on these suites, but they come from the same authors and judges as SelfJev's training data, so they favour SelfJev.
• SelfJev is the most accurate open model at 4.5B parameters or less on both suites (paired tests, p ≤ 0.023) and ties imajev-4b on held-out images; the 27B openjev is higher on Text Decisions. Among the 4B models only jpt-4b is faster on Text Decisions, and kev-4b and jpt-4b on AI Response Review.

Protocol, per-model setups, paired tests and raw reports: reports/competitors.

Public datasets

83.4% across 3,300 questions. There are 300 questions per source, so the macro average over sources and the question-weighted average are equal. The 171 authored development questions are excluded here; the full 3,471-question development benchmark remains available in the reports.

“Held-out source” means that dataset was excluded from SelfJev's task-specific training. “Seen source” means other rows from that source were used in training. Neither establishes absence from Qwen's pretraining.

These are adapted tasks, not official dataset leaderboard scores. For example, Banking77 uses eight candidate options per question, CLINC uses seven or eight, and DBpedia uses six. MNLI is recast as binary entailment. GoEmotions scores exact matching over six candidate tags. Our preprocessing, sampling and answer sets are part of the evaluation and must accompany the numbers.

Known development-data issues include shared MNLI premises across splits and near-duplicate authored policy texts. See data methodology. All 11 slices are shown, including weaker emotion and sentiment results.

Download chart data · SelfJev predictions · Matched Jev predictions

Accuracy and model size

All points use the same 3,300 question IDs, labels and candidate sets. The stock models are rerankers mapped to our decision tasks. The earlier LoRA runs use older training recipes; SelfJev uses a different backbone generation and more training data. This shows the measured systems' trade-off between size and accuracy. It does not isolate the effect of parameter count or establish that a 4B model outperforms larger models generally.

Sizes are nominal backbone parameter counts, not checkpoint megabytes or runtime memory. Jev appears as a horizontal reference because its parameter count is undisclosed. We do not assign a guessed size to proprietary models.

AI response review

Breakdown of the 946-question authored suite, using the same current-engine predictions as the overview:

Labels were authored and checked with AI judges. Questions about one document are correlated; small differences should not be read as proof of superiority. This suite measures our defined answer choices and criteria, not unrestricted judgment quality. Dataset design · Predictions

Quickstart

Use a Linux NVIDIA GPU machine with a compatible vLLM Python environment, Git and pip. Install SelfJev into that environment and download the merged checkpoint:

GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/Jwuthri/SelfJev.git
cd SelfJev
# Run inside your vLLM environment; it must support Qwen3.5.
python -m pip install -e '.[serve]'
python - <<'PYTHON'
from huggingface_hub import snapshot_download
snapshot_download(
    "Jwuthrich/selfjev-4b-merged",
    local_dir="weights/selfjev_4b_merged",
    ignore_patterns=["assets/*", "reproduce/*"],
)
PYTHON

export SELFJEV_API_KEYS="replace-with-your-long-random-secret"
selfjev serve --engine vllm \
  --model-dir weights/selfjev_4b_merged --host 127.0.0.1 --port 8000

This uses SelfJev's vLLM backend. The default TreeServer quickstart uses the separate adapter release. A generic chat endpoint does not implement SelfJev's decision scoring.

In another terminal, use the same secret:

from selfjev import SelfJev, Noul, Choice

client = SelfJev(
    base_url="http://127.0.0.1:8000",
    api_key="replace-with-your-long-random-secret",
)
result = client.system_one(
    state="I was charged twice. Please refund the duplicate payment.",
    questions={
        "refund": Noul("Does the customer want a refund?"),
        "team": Choice("Which team should handle this?", {
            "billing": "payments and refunds",
            "support": "technical issues",
        }),
    },
)
print(result.nouls["refund"].noul)
print(result.choices["team"].choice)

The example shows the interface; it does not present fabricated model output. A generic Transformers text-generation or classification pipeline does not reproduce SelfJev's prompts, scoring rules or API. The server key is a secret you choose for your deployment, unrelated to Hugging Face credentials.

HTTP API · Self-hosting and AWS · Fine-tuning

Architecture

The default engine builds a shared-prefix tree: compute the document once, branch for each question, then branch for candidate answers. It reads yes/no evidence from the logits and converts those scores into the requested answer type. The server includes every candidate in the question, matching training.

The tree saves repeated computation at two levels: every question reuses the document, and every candidate reuses its question. In full-attention layers, the tree mask lets a branch read its ancestors but not sibling branches. In DeltaNet layers, branches inherit the recurrent state of their parent. Each candidate therefore sees its own document → question → candidate path, up to numerical differences between kernels. The diagram describes TreeServer; vLLM uses its own prefix-cache execution.

Jev probabilities served as a teacher signal; they did not decide the authored training labels. Authored labels were checked by an AI judge. Full lineage is recorded in model.json, with architecture detail in the research documentation.

The research path

The improvements came from several changes: verified difficult examples, the instruct backbone, explicit option lists, broader training coverage, tree-aware training and soft teacher targets. The chart is a history of complete systems, not an ablation assigning each gain to one change.

The research also includes smaller stock rerankers, 8B models, Jina, T5Gemma, custom architectures, distillation and RLCD. The full experiment ledger records results and dead ends; earlier implementations remain at the archive tag documented there. The checkpoint's original engine recorded a slightly different text-decision score; this card consistently uses the current engine for SelfJev's headline results.

Hardware and speed

A 24 GB NVIDIA GPU is a practical starting recommendation, not a measured minimum. Plan for 16–32 GB host RAM and at least 50 GB free disk. The full backbone and runtime state must fit in memory; checkpoint file size alone does not determine runtime requirements.

The default engine avoids token-by-token answer generation and shares work across questions. Latency still depends on input length, question count, candidate count, GPU and concurrency. We have two sets of measurements below, with different models and engines. Current SelfJev-4B has no controlled NVIDIA latency sweep yet. Its CPU minimum is also unmeasured.

Earlier prototype on NVIDIA GPUs

These are server-side medians, excluding network time: ten warmed, successful requests per point, one request at a time, three answer options per question. All three sweeps use the archived tree_4b_combo Qwen3-4B model on vLLM in bf16. They were recorded in separate runs, not a simultaneous controlled hardware experiment. The plot uses a labeled logarithmic time axis; the table gives the actual milliseconds.

These measurements show what the earlier system achieved; they are not a latency promise for the current Qwen3.5 model or the merged release. A10G samples · L40S samples · H100 samples

Current SelfJev-4B on Apple Silicon

This uses the current adapter merged into Qwen3.5-4B, the TreeServer engine, and PyTorch MPS in bf16 on a 20-core Apple M5 Pro GPU with 48 GB unified memory. Each cell is the median of ten timed calls after two warmups, with MPS synchronized before and after each call. Inputs are synthetic repeated text, with three answer options per question. Timing includes tokenization and model work; it excludes HTTP/network.

The MPS run uses the slower PyTorch recurrent fallback. It is a lower-level engine experiment, not a validated Mac server. Model and engine differences prevent a hardware-only comparison against the NVIDIA chart. The merged download also has not been benchmarked through vLLM using these workloads.

Download latency samples, configuration and medians · Latency CSV · Full speed research

Limits and reproducibility

• Probability is not a guarantee. The reported current-engine runs have no fitted calibration attached. Validate thresholds on application data before acting on confidence.
• Some tasks remain weak. Emotion labeling, fine-grained sentiment and selecting an exact set of labels are visibly harder than topic and intent classification.
• Benchmarks have a scope. AI-authored tests, reused development sets, sampled options and known overlap issues limit the conclusions. We make no general state-of-the-art claim.
• Engine versions matter. These accuracy results refer to the recorded current TreeServer runs. A merged checkpoint, quantization or another serving backend needs its own verification.

Every plotted value is regenerated from saved predictions or raw timing samples. The generator checks matched IDs, targets, question types and candidate sets for report-to-report comparisons, and recomputes latency medians from ten timed samples per cell. Chart data and source SHA-256 hashes · Generator · Architecture and latency plotting module · Card template

Place the downloadable generator, plotting module and template in scripts/docs/ of a SelfJev source checkout, then run:

uv run --no-project --with matplotlib==3.11.1 python scripts/docs/build_model_card.py

This builds the shared figures and adapter card from reports and the canonical data/all.jsonl.gz; it performs no model inference. The merged-card converter reuses those files, updates serving details and refreshes the existing checksum manifest. Run it from scripts/docs/ with --checksums pointing to this release's downloaded SHA256SUMS. Rebuild the canonical file using the data instructions if absent.

Source adapter SHA-256: dfbf2834d883987893ec305a6093a345fd79c77f903f60f2f914cbe3b6058d1b.

The base model's license is retained. No separate license for the SelfJev fine-tuning contribution has been declared. Independent project; not affiliated with TypeSafe or Qwen.