amad-vlm6
amad-vlm6 is a TIES merge of two Arabic OCR fine-tunes of Qwen2.5-VL-7B-Instruct: amad-iq/amad-vlm5 and AhmedZaky1/DIMI-Arabic-OCR-V2.
📄 Full technical report — method, per-dataset results, and an error-type decomposition showing what the merge actually fixed.
Results
KITAB-Bench ocr-eval, 13 datasets, 3,760 images, final-answer scoring with the
benchmark's unmodified metrics.
| amad-vlm5 | amad-vlm6 | |
|---|---|---|
| CER ↓ (macro) | 0.254 | 0.235 |
| CHrF ↑ (macro) | 81.05 | 81.68 |
| CER ↓ (10 leak-free datasets) | 0.321 | 0.236 |
| CHrF ↑ (10 leak-free datasets) | 77.35 | 82.46 |
Largest per-dataset gains: khattparagraph CER 0.711 → 0.088, evarest 0.291 → 0.142, hindawi 0.241 → 0.137.
What the gain is. Error-type decomposition (see the report) shows the improvement is behavioral: the merge cuts amad-vlm5's thinking rate on page-level inputs from 45–98% to 5–18% and removes its truncated and empty outputs, so insertions and deletions collapse. Reading accuracy itself (substitution rate) is worse on 10 of 13 datasets and better only on adab and evarest. Hallucination is not cancelled but moved: on isippt and muharaf the merged model over-generates where amad-vlm5 did not. Four per-dataset wins and three losses are statistically significant (bootstrap); five are not.
Files
| Repository | Format | Use |
|---|---|---|
| this repo | bf16 safetensors | Transformers, vLLM, further fine-tuning |
| amad-vlm6-GGUF | Q8_0 + mmproj | llama.cpp, LM Studio |
Usage
import re, torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
repo = "amad-iq/amad-vlm6"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
repo, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)
messages = [{"role": "user", "content": [
{"type": "image", "image": "page.png"},
{"type": "text", "text": "Extract the text in the image. Give me the final text, nothing else."}]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4096, do_sample=False, repetition_penalty=1.05)
raw = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(re.sub(r"<think>.*?</think>", "", raw, flags=re.S).strip())
The model inherits a thinking behaviour from one parent: on dense pages it may
emit <think>…</think> before the transcription. Keep only the text after the
last </think>, and allow 4,096 new tokens — a 500-token budget truncates
dense pages.
Limitations
- Reading regresses on 10 of 13 datasets (substitution rate); the CER wins come from output discipline, not perception. Macro WER is worse (0.361 → 0.463), driven by three datasets with catastrophic rows (historyar, muharaf, isippt), not by a general segmentation problem.
- Two large regressions from imported over-generation: isippt CER 0.054 → 0.541 (insertions 0.006 → 0.471, 500 images) and muharaf 0.046 → 0.600 (insertions 0.012 → 0.423, plus memorized reading lost). Avoid on presentation-slide and muharaf-style handwritten inputs. onlinekhatt and khatt (amad-vlm5's other training-overlap sets) also regress.
- Not yet shown to beat the cheaper fix. The natural control — amad-vlm5 with a repetition penalty and a think budget at inference — is being run; until it lands, merging is not proven necessary for this gain.
- One benchmark is a narrow window. These numbers cover transcription accuracy on 13 Arabic document styles. They say nothing about photographed pages in poor light, skew, mixed Arabic–Latin text, tables and layout, latency, or instruction following — where this model may hold unmeasured strengths and unmeasured regressions.
Reproduction
merge_method: ties
base_model: Qwen/Qwen2.5-VL-7B-Instruct
models:
- model: amad-iq/amad-vlm5
parameters: {weight: 1.0, density: 0.5}
- model: AhmedZaky1/DIMI-Arabic-OCR-V2 # V2 only — V1 is subsumed by V2
parameters: {weight: 1.0, density: 0.5}
parameters: {lambda: 1.0, normalize: false}
dtype: bfloat16
License
Apache 2.0, inherited from the base model and both parents.