instinct-vision-san
A 500M vision-language model finetuned for live camera temporal reasoning — change detection, change direction, and temporal order binding — built on the laya/jev scoring architecture (single forward-pass option scoring, no autoregressive generation). Trained without any restricted-license data, making it usable as a commercially-distributable checkpoint, and small enough for edge/NPU deployment.
Highlights
- 500M parameters (SmolVLM2-500M-Video-Instruct backbone, Apache 2.0) — ~40% smaller than instinct-vision-ni while matching its temporal competency
- Single forward-pass option scoring — no autoregressive decode; scores answer options in one pass
- Multiframe temporal reasoning: 6/6 held-out jitter suites @1.000, stable across seeds (999/42/43)
- ScienceQA image-only 85.3% (N=143) — slightly above the 0.8B ni model
- ~25-frame context fits
max_len2048 thanks to SmolVLM2's token-efficient video encoding - Fast: 0.052 s 1-frame / 0.081 s 2-frame (GB10, bf16)
- Edge-validated: RKNN fp16 port on RK3588 showed 10/10 agreement with the bf16 host model
Results
| Test | Result |
|---|---|
| Multiframe temporal, held-out jitter N=50 × 6 suites | 1.000 each (6/6 @1.000), stable across seeds 999/42/43 |
| ScienceQA image-only (N=143) | 85.3% |
| Speed, 1-frame / 2-frame (GB10, bf16) | 0.052 s / 0.081 s |
| COCO general VQA (519 auto-generated MCQs, eval only) | 83.0% overall |
| — count | 33.7% |
| — attribute | 98.0% |
Comparison with instinct-vision-ni (0.8B, same clean-data recipe):
| Test | ni (0.8B) | san (this model, 500M) |
|---|---|---|
| Multiframe temporal N=50 × 6 suites | 1.000 each | 1.000 each |
| ScienceQA image-only N=143 | 84.6% | 85.3% |
| Speed 1-frame / 2-frame | 0.056 / 0.097 s | 0.052 / 0.081 s |
san matches ni on every temporal suite, slightly beats it on the ScienceQA spot-check and on attribute/VQA-style general questions, at ~60% of the parameter count.
Honest caveat: counting/numeric QA is weak (~34% on COCO count questions). For counting tasks, pair this model with an object detector.
Training data (full disclosure)
| Source | Examples | License | Notes |
|---|---|---|---|
| ScienceQA (derek-thomas/ScienceQA mirror, train split) | 5,000 | CC BY-SA 4.0 | general VQA retention |
| AI2D (lmms-lab-encoder/ai2d, test split, first 250 rows excluded) | 2,821 | CC BY-SA 4.0 (mirror card carries no explicit tag — caveat noted) | diagram QA; ⚠️ future AI2D evals on this split are contaminated for this model |
| Synthetic temporal pairs from own camera snapshots (PIL jitter, 896px) | 5,000 | self-made photos, Apache-2.0 use | delta/same/which × pair-type × polarity grid; private frames used at train time only, never shipped |
| Total | 12,821 |
Excluded by design: A-OKVQA, VQAv2, any COCO-image training set (COCO used for evaluation only), iconqa (NC), chartqa (GPL), VLFeedback/RichHF/CrisisMMD (NC).
Architecture & training
- Backbone: HuggingFaceTB/SmolVLM2-500M-Video-Instruct (Apache 2.0), natively video/multi-frame
- laya-vision scoring head, full-model finetune (head-only freeze was tried first and fails temporal order binding on this backbone: which-reversed 0/50 — the backbone must be finetuned at low lr to bind frame order)
- lr 1e-4 head / 1e-5 backbone, batch size 8,
max_len2048, AdamW8bit, 4,800 steps (~3,100s on GB10) - Post-loop per-type temperature calibration (filtered 3.5k mix)
Usage
Requires the laya-vision package. See example.py for a runnable snippet.
from laya import VLMAgent
model = VLMAgent("spele1100/instinct-vision-san", device="cuda", dtype="bf16")
# Single-frame QA
result = model.score({"images": ["frame.jpg"], "prompt": "What changed?"})
# Multi-frame temporal reasoning — frames in temporal order
result = model.score({"images": ["f1.jpg", "f2.jpg"], "prompt": "Did the object appear or disappear?"})
State dict accepts {"images": [...]} for multi-frame input; ~25 frames fit max_len 2048.
License
Apache 2.0 (this checkpoint + code). Training-data licenses disclosed above; ScienceQA and AI2D are CC BY-SA 4.0 (share-alike applies to those datasets). No COCO-derived images were used at train time.