rsi-jev-v4.0-vl-qwen3.5-2b

A typed-decision model: Qwen/Qwen3.5-2B-Base with its tower fine-tuned and a trained option-scoring head. It answers choice / noul / score questions about a document and up to four images in one forward pass, and speaks the Jev HTTP API.

Images need the vision extra (Pillow, torchvision): pip install "rsi-jev[vision] @ git+https://github.com/Shanghua-Gao/RSI-Jev". The vision encoder and image processor are read from the base model.

This checkpoint (seed 17, one seed):

Re-computed from these exact files after upload-time reload. Kernel stack fla-0.5.2/torch-2.7.1+cu128; full recipe in meta.json.

import sys; sys.path.insert(0, 'code')   # after snapshot_download of this repo
from load_release import load_release
model, tok, enc, meta = load_release('.', device='cuda')
print(meta['calibration'])   # the fitted calibration this checkpoint ships, or 'none'

load_release applies calibration.safetensors when the checkpoint carries one, so the probabilities are the calibrated ones this card reports. A loader that skips those files returns the raw distribution instead: the same chosen option, different numbers.

Everything below is the version record from the project repository, unchanged.

RSI-Jev v4.0-VL

Describes release v4.0-VL · updated 2026-09-30

RSI-Jev now reads images. Send one to four pictures with a request — a photo, a screenshot, a scanned page, a chart — and ask typed questions about them: yes/no, pick-one-of-k, rate-on-a-rubric. As before, one forward pass returns a calibrated probability for every option, with no generated text. Text requests work exactly as they did.

On five image benchmarks held out from training it answers 80.3% of questions correctly; with the images blanked out it drops to 45.8%, so the answers come from the pictures. On text it keeps v3.0's level: 0.756 on the fifteen-benchmark suite, 0.653 on the held-out set.

v4.0-VL is v3.0's lineage with three rounds of image training, a calibration data round, and a new RL stage on top. It is a 2B model, one seed.

1. The task

Unchanged from v1.0, plus images: a document (the state), now optionally with 1–4 images placed in it by <image> markers, and one or more typed questions — noul, choice, score — with a probability per option, returned in one forward pass. Top-1 is the fraction of decisions whose highest-probability option matches the label; ECE is expected calibration error, the average gap between the confidence a model reports and how often it is right at that confidence, lower being better.

2. How the model is built and trained

Qwen3.5-2B-Base, with v1.0's option scorer, v2.0's confidence head and v3.0's MLP readout. Images go through the base model's own vision tower, frozen: its weights are not in this checkpoint and are loaded from Qwen/Qwen3.5-2B-Base. Every request gets at most 1,024 image tokens per question, split evenly across its images.

The stages, each continued from the one before:

1. Text, supervised. v3.0's data-scale-2 corpus with the Jev-Style sources capped at 3,000 questions each, 14,233 steps.
2. Images, supervised, 4,000 steps. 33,906 image questions from the train splits of twelve public datasets (DocVQA, ChartQA, VQAv2, A-OKVQA, NLVR2, Hateful Memes, Screen2Words, AI2D, ScienceQA, TallyQA, RVL-CDIP, TQA), mixed with 30,094 questions of text replay. Free-form answers became four-way choices with real distractors, or balanced yes/no questions.
3. v3.0's listwise reranking RL stage, run on the image model.
4. Images, second round, 3,000 steps. 17,352 new image questions — more public VQA train sets and six synthetic generators (UI state, change detection, line rules, grid games and others) — with 51% text replay.
5. Calibration data, 3,000 steps. 8,492 generated reasoning questions (lie chains, object swaps, orderings) at depths where the model is at chance, trained toward soft targets so it learns to say so. These share task families with BIG-Bench Hard; see §4.1.
6. An asymmetric RL stage, 1,500 steps. The reward is correctness minus a confidence penalty that charges a confident mistake four times what it charges a timid correct answer: R = c − 4·max(0, p − c)² − 1·max(0, c − p)², with p the probability on the chosen option and c whether it was right. 30% of its rows carry images. It calibrates in the weights: held-out ECE before any calibration falls from 0.200 to 0.082.

Against a matched supervised stage — the same data, steps and seed, with only the objective changed — the asym stage gave better held-out accuracy and calibration, with no detectable ranking difference (one seed):

The AUROC gap is −0.008 with a paired bootstrap standard error of 0.006.

Calibration. The confidence head (oof_head_scorefloor, as in v2.0–v3.0) is refitted on 8,464 held-out questions: one in ten of the text corpus, plus 1,251 image questions weighted 0.15. Cases whose document or image also appears in a trained case were dropped first. It rescales each question's probabilities without changing which option wins.

3. Checkpoints

v4.0-VL and v3.0 were scored in one job by the same evaluator; the v2.1 and v1.0 rows are from v3.0's record, on the same evaluator, and v3.0 reproduces its own published numbers exactly in this job. 2B only, one seed's checkpoint.

Reproducing its training run. Re-scored with this repo's loader, the checkpoint agrees with its training record on every question: typed-decisions 0.787 and 0.7835 (reversed order), MMLU-Pro 0.385, agreement 1.0 on each (verify.json). Reading a document once and answering every question from it gives the same numbers as reading it per question: on CPU in fp32 the largest difference is 2e-6 over 100 questions (cached_agreement.json); on the GPU the fp32 kernels leave up to 5e-4 and no answer changes over 500. The served image path matches the offline image encoder on all 50 image items. Reloaded from the saved files with calibration applied, the checkpoint agrees with the in-memory model on 200 of 200 text questions (largest probability difference 0.0005) and 50 of 50 image questions (roundtrip_200.json). On a second GPU architecture (Blackwell), this repo's image path and the training code agree with each other to 1e-7 and differ from the first GPU on one of the 50 image questions, whose top two options sit at 0.376 and 0.400: the same order-of-summation effect v2.1 §3 describes for text. MANIFEST.json holds the sha256 of every file.

The checkpoint ships calibration.safetensors and calibration.json; load_release.py applies them automatically. Without them the model loads uncalibrated and its probabilities match no number in this card.

Apache-2.0, following the base model, as every release is.

4. Results

4.1 The held-out comparison — read this first

eval_final_v2, as in v3.0: LegalBench (11 tasks), MuSR, BIG-Bench Hard (5 tasks), the relevance split of the Berkeley Function-Calling Leaderboard, and ConditionalQA, built from sources in no training corpus.

Both models were scored in one job, post-calibration. p is an exact two-sided McNemar test on the questions where the two disagree (pooled: 258 won by v4.0-VL, 244 by v3.0). BIG-Bench Hard is not held out for this release: stage 5's generated questions share its task families, so the row without it is the held-out number. Held-out ECE is 0.082.

4.2 The fifteen-benchmark suite

Ten of the fifteen benchmarks contribute training data in some form, so none of these numbers is zero-shot; §4.1 is the comparison that is.

4.3 Images

Held out from training. Five public benchmarks, 4,173 questions, equal weights. Their images were checked against every training image (sha256 and perceptual hash) and their text against the training text; InfographicVQA was kept out of training entirely. "Blank" replaces every image with a blank one and asks the same questions.

The confidence head brings image ECE from 0.120 to 0.069 without changing any answer. To score these yourself: scripts/vision_benches/heldout.py.

Other public benchmarks, not checked against the training images, run as served (bf16 tower, calibration on):

VisA is zero-shot: the model is shown one image and asked whether the part is defective, and has seen no examples of normal parts. The score averages both option orders. To re-run all three, with the paired comparisons: scripts/vision_benches/.

Laya Vision's validation set (6,357 questions over 34 sets), against Laya Vision's own calibrated predictions on the same items. "Clean sets" drops 14 whole sets that overlap this release's training data, among them IconQA, Visual7W and OCR-VQA, which it trained on.

It loses on the rating sets: aesthetic scores on AVA (0.140 against 0.845), CrisisMMD, RichHF and VLFeedback. Calibrated ECE is lower than Laya Vision's on every pool (0.061 against 0.087 on the clean sets). Re-run: scripts/vision_benches/laya.py.

The demo gallery. 143 pictures and 227 questions across road scenes, "is it there?", counting, charts, screens and factory parts: 187 answered right (82%), median 62 ms per call on the GB10. Every miss stays in the gallery. It also plays Breakout from pixels: 10 of 10 on the game's lane check, and in a live game it cleared 7 of 24 bricks without losing a life before the game's 200-decision limit. Re-run: examples/gallery/ (every image ships with it) and examples/breakout/.

4.4 Calibration and the properties we kept

Suite ECE after the confidence head is 0.043 (0.052 before it). Three properties external testing found useful were checked with the same scripts, in the same job as v3.0:

5. Speed, the wire format, and what confidence means

Text inference is unchanged: one tower pass per question, the calibration a division inside the readout, nothing generated.

Images (../serve/README.md has the limits):

• A request carries "images": ["data:image/png;base64,…"] beside state: 1–4 PNG, JPEG or WebP data URLs. The state marks where each goes with <image>; without markers the images go first. URLs are not fetched.
• The vision tower runs once per request, and the state with its image tokens is read once for all the questions. With --profile agent, a repeated image state is not read again.
• Up to 1,024 image tokens per question (about one megapixel for a single image, 256 tokens each for four) on top of the 2,048 text tokens.
• On the GB10, one 640 px photo takes 80 ms with one question and 110 ms with four; a 1,600 px photo, 245 ms and 273 ms. Asking again about the same image under --profile agent takes 36–101 ms. ../docs/inference.md has the table.
• Image requests need Pillow and torchvision: pip install "rsi-jev[vision]". Without them the server still starts and answers text; an image request gets a 422 saying what to install.

A structured criterion (an object rather than a string) is still rejected with 422, and a document longer than 2,048 tokens is cut from its start. With images, a request whose cut would reach an image is refused rather than cut.

The API returns a probability and a confidence, and they are different numbers: confidence is the peak statistic Jev documents, (K * p_max - 1) / (K - 1). Route on the probability of the option you intend to act on.

6. Limitations

• Images are capped at four per request and 1,024 tokens per question. A large photo is scaled down to fit, so small text in a full-page scan can be lost.
• Image requests are slower than text: an image adds hundreds of tokens, and a large photo up to 1,024 (§5).
• The vision tower is the base model's, unchanged. Everything this release learned about images is in the text tower and the scorer.
• BIG-Bench Hard is not held out for this release (§4.1).
• The weaknesses external testing found in v2.1 are still here: a lean toward the more severe option on ordered choices, rarely choosing "unknown" when a question cannot be answered from the document, and inconsistent answers to a question and its negation. Do not use it where an "unknown" option has to be honoured without checking it on your own data.
• The recipe has not been retrained from this repo. Its code, stage specs and data builders are here (§7). The stage corpora and the synthetic image questions were rebuilt with them to the same bytes (the rendered images differ slightly with font versions); the public image corpora need their downloads and were not rebuilt.
• Documents over 2,048 tokens lose their beginning (§5).
• One model, one size, one seed. The comparison with the matched supervised stage is one seed too.

7. How it got here

Rejected along the way: a second image data portfolio that raised its own probes but not the held-out image set (−0.006), and training the calibration round on true answers instead of soft targets (it learned the reasoning families and failed held-out calibration). The full record is ../EXPLORE.md.

Rerunning it

Each stage's spec is in ../data/v4.0-vl_recipe/; asym-calA.json is this checkpoint's own. Paths in them are relative to one data directory, D below. The data steps run on CPU and download their public sources; nothing is shipped.

A fix that also applies to v2.1 and v3.0. Until this release, this repo's fit.py held the lower layers' learning rate constant after warm-up (lower_layers_n), where the code that trained v2.1, v3.0 and this release's supervised stages decayed them with the rest of the tower. It now matches that code, so those recipes rerun as they were trained. The RL stages were never affected.

# image corpora (vision_v3 is generated; it needs fonts, see the script)
python scripts/build_vision_v1.py --root D/vision_v1
python scripts/build_eval_vision.py --build D/eval_build
python scripts/decontam_vision.py eval --build D/eval_build --train D/vision_v1 --out D/eval_vision_v1
python scripts/build_vision_v2.py --root D/vision_v2 --v1 D/vision_v1 --eval-build D/eval_build
python scripts/decontam_vision.py v2 --root D/vision_v2 --v1 D/vision_v1 --eval-build D/eval_build --eval D/eval_vision_v1
python scripts/build_vision_v3.py --root D/vision_v3 --font-dir FONTS
python scripts/decontam_vision.py v3 --root D/vision_v3 --eval-build D/eval_build

# stage corpora; V3 is v3.0's SFT corpus and RL2 its RL file (versions/v3.0.md)
B="python scripts/build_stage_corpora.py"
$B rep-b    --root D --ds2 V3
$B vis-ct-1 --root D --vision-v1 D/vision_v1
$B rl5-vis  --root D --rl-hrr RL2/rl_hrr.jsonl
$B vis-ct-3 --root D --vision-v2 D/vision_v2 --vision-v3 D/vision_v3
python scripts/build_depthdial.py folds --folds D/depthdial/folds
python scripts/build_depthdial.py build --folds D/depthdial/folds \
    --cells data/depthdial_cells_vis-ct-3.json --parent vis-ct-3 --out D/depthdial/vis-ct-3
$B calA-ce  --root D --dd D/depthdial/vis-ct-3
$B rl2-b2   --root D --ds2 V3
$B calasym-stage --root D

# the six stages, each continuing from the one before (one GPU each; the image
# stages need an 80 GB card)
for s in v4-rep-b-s17ck vis-ct-1 rl5-listwise-vis vis-ct-3 calA-ce asym-calA; do
  c=$(python -c "import json;print(json.load(open('data/v4.0-vl_recipe/$s.json'))['corpus_dir'])")
  python scripts/release_train.py --model Qwen/Qwen3.5-2B-Base --seed 17 --root D \
      --spec data/v4.0-vl_recipe/$s.json --corpus D/$c --save-dir D/$s/s17 --out D/runs --name $s
done

# the confidence head: one in ten of stage 1's corpus plus the vision_v1 holdout
python scripts/fit_release_calibration.py --ckpt D/asym-calA/s17 --root D --lineage \
    --dev-corpus D/v4-rep-b-s17ck \
    --dev-sources "$(python -c "import json;print(json.load(open('data/v4.0-vl_recipe/v4-rep-b-s17ck.json'))['sources'])")" \
    --vis-dev-root D/vision_v1 --out D/asym-calA/s17

data/depthdial_cells_vis-ct-3.json is vis-ct-3's measured accuracy per task and depth, from which the calibration round's soft targets are set; scripts/depthdial_measure.py remeasures it from a vis-ct-3 checkpoint.

Image sources and their licences. The images are the original datasets', under their own terms; each builder's docstring records the exact revision. Of vision_v2, only the four sources below were trained on. Five sources are non-commercial or research-only, and several others carry no clear grant for their images:

The weights are released under Apache-2.0, as earlier releases were. Whether weights trained on non-commercial data inherit those terms is not settled; check the sources above against your use.