InSight-doc-8B
📄 Paper | 💻 Code | 🧩 SFT Data | 🎯 RL Data | 🎬 Replay Demo | 🚀 Live Demo
Overview
InSight-doc-8B is an 8B-class vision-language agent for long-document understanding. It is initialized from Qwen/Qwen3-VL-8B-Instruct and trained to use visual resolution as an adaptive inference-time resource: start from low-resolution page views, call a zoom-in tool on selected regions, then answer from the returned high-resolution evidence.
This checkpoint is a Hugging Face-compatible Qwen3-VL model. To reproduce the
reported agentic behavior, use it with the InSight-doc agent loop and
image_zoom_in_tool rather than plain single-turn inference.
Model Details
| Item | Value |
|---|---|
| Model | InSight-doc-8B |
| Base model | Qwen/Qwen3-VL-8B-Instruct |
| Architecture | Qwen3-VL vision-language model with tool-use SFT and RL |
| Primary task | Long-document visual question answering |
| Agent tool | image_zoom_in_tool for region-level visual evidence acquisition |
| Training data | InSight-doc-SFT-18k and InSight-doc-RL-19k |
| License | Apache-2.0 |
Training Summary
InSight-doc-8B is trained in two stages:
- Supervised fine-tuning. Full-parameter SFT from Qwen3-VL-8B-Instruct on
17,913 curated zoom-in trajectories. The released recipe freezes the vision
tower, uses max sequence length 65,536, sequence parallel size 4, global
batch size 32, cosine LR
5e-6 -> 5e-7, and two epochs. - Reinforcement learning. RL starts from the SFT checkpoint and trains on 19,236 hard prompts with the same zoom-in agent interface. The released RL recipe uses weighted refill source sampling, temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5, and 2,000 total RL steps.
The SFT data contains 14,216 answerable and 3,697 unanswerable trajectories. The RL data contains 10,579 answerable and 8,657 unanswerable prompts, with the released sampler targeting an effective 86%/14% answerable/unanswerable mix.
Evaluation Highlights
Across DUDE, MP-DocVQA, MMLongBench-Doc, and LongDocURL, InSight-doc-8B improves over Qwen3-VL-8B by 4.3-16.4 accuracy points under medium-to-low input resolution settings. On MMLongBench-Doc and LongDocURL, it reduces hallucination on unanswerable questions by 40%+ and lowers latency by 41%-68% (1.7x-3.1x speedup) while maintaining an accuracy lead.
Across the four document VQA benchmarks, InSight-doc shifts the accuracy-efficiency Pareto frontier upward and leftward, achieving higher accuracy with shorter sequences and lower latency. Darker points indicate higher initial input DPI.
See the paper and release repository for complete benchmark tables and the exact evaluation configuration.
Demos
- Replay demo: compares InSight-doc-8B and Qwen3-VL-8B side by side on replayed examples.
- Live demo: runs an interactive InSight-doc-8B demo.
Usage
Agentic Evaluation
The recommended path is the release evaluation script, which serves the model with vLLM and runs the shared InSight-doc agent/evaluator:
git clone --recurse-submodules https://github.com/m-Just/InSight-doc.git
cd InSight-doc
pip install -e .
pip install -e ./verl
export MODEL_PATH=InSight-doc/InSight-doc-8B
export VAL_FILES='/path/to/longdocurl.parquet,/path/to/mmlongbench.parquet'
export RESCALES='0.25 0.35 0.5'
export EVAL_CUDA_VISIBLE_DEVICES=0,1,2,3
export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://.../v1
bash scripts/evaluate_insight_doc.sh
Important vLLM defaults in the release config are 4 replicas, 1 GPU per
replica, max_model_len=262144, max_num_seqs=64, chunked prefill, prefix
caching, and the same sampling settings used by RL validation.
Plain Transformers Loading
The checkpoint can also be loaded as a regular Qwen3-VL model for basic experimentation, but this does not execute the zoom-in tool loop:
import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
model_id = "InSight-doc/InSight-doc-8B"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
Use explicit bf16 loading for inference; the uploaded config currently records
dtype: float32, so dtype="auto" may load fp32 weights in plain Transformers.
Input Format
The agentic policy expects multimodal chat messages containing low-resolution
page images and a tool schema for image_zoom_in_tool. During generation, the
model may emit structured tool calls that specify an image index and a 2D
bounding box. The runtime crops the requested region from the original page and
returns it as a new visual observation.
The released SFT dataset includes the tool schema in a tools column. For RL
and evaluation, the same schema is loaded from the release configuration at
recipe/vsearch/config/image_zoom_in_tool_config.yaml.
License
The model weights are released under the Apache License 2.0. The checkpoint is derived from Qwen/Qwen3-VL-8B-Instruct, which is also released under Apache-2.0.
Citation
@article{li2026insightdoc,
title={InSight-doc: Agentic Visual Perception for Long-Document Understanding},
author={Li, Kaican and Xie, Weiyan and Yao, Lewei and Wu, Jiannan and Hong, Lanqing and Huang, Yongxiang and Zhang, Nevin L.},
journal={arXiv preprint arXiv:2608.10628},
year={2026}
}