Qwen2-VL-7B: Specialized ACORD Insurance Form Extractor
Engineering Overview
This model is a state-of-the-art Vision-Language Intelligence (V-LIE) engine, fine-tuned from Qwen2-VL-7B-Instruct. It is architected to eliminate traditional multi-stage OCR pipelines by mapping pixels directly to structured semantic JSON for complex, multi-column insurance documents.
Technical Specifications
- Base Architecture: Qwen2-VL (ViT + Qwen2 LLM) utilizing Multi-Modal Rotary Positional Embeddings (M-RoPE) for spatial reasoning.
- Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) via QLoRA (4-bit NormalFloat quantization).
- Adaptation Strategy: LoRA target modules included
q_proj,k_proj,v_proj,o_proj, and MLP layers (gate_proj,up_proj,down_proj). The vision encoder remained frozen to preserve generalized OCR capabilities while the language head was adapted for schema-strict JSON generation. - Resolution Handling: Supports dynamic resolution (256-1280 pixels) to maintain aspect ratio integrity, crucial for identifying dense ACORD form checkboxes and fine-print labels.
- Training Objective: Supervised Fine-Tuning (SFT) using a custom data collator to mask prompt/image tokens, focusing the cross-entropy loss calculation exclusively on the assistant's JSON output tokens.
Optimized Inference Pipeline
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image
# Model weights are merged into BF16 for inference stability
model_name = "solvrays/scribegene-llm-v1.2"
model = Qwen2VLForConditionalGeneration.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto",
attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained(model_name)
def extract_acord(image_path):
image = Image.open(image_path).convert("RGB")
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image, "min_pixels": 256*28*28, "max_pixels": 1280*28*28},
{"type": "text", "text": "Extract the structured field JSON for this ACORD form page."},
],
}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)
# Use greedy decoding for deterministic JSON output
output_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
trimmed = output_ids[:, inputs["input_ids"].shape[1]:]
return processor.batch_decode(trimmed, skip_special_tokens=True)[0]
print(extract_acord("sample_acord_125.png"))
Key Advantages vs. LayoutLMv3
- Unified Architecture: No need for external OCR (Tesseract/Textract) or bounding box pre-processing.
- Generative Flexibility: Handles non-standard field layouts and handwritten text variations that often break token-classification models.
- Multi-Page Context: The native image-text interleaving allows for potential extension into multi-page document reasoning in a single context window.