👁️ PaliGemma-BlindAssist: A parameter-efficient vision-language model adapter for blind assistance VQA & Captioning using LoRA
🎯 Model Overview
A full-precision (bfloat16) adaptation of
google/paligemma-3b-mix-224
for accessibility vision-language tasks, trained with LoRA on the VizWiz
datasets and merged into a standalone checkpoint. It performs visual question
answering and scene captioning for blind and low-vision users, and represents
the quality ceiling of our adaptation: the
4-bit QLoRA variant
matches it to within a fraction of a point at 40% of the memory footprint, and
is the variant intended for on-device deployment.
Adaptation improves the target task substantially while leaving general-domain performance largely intact. On VizWiz captioning, CIDEr-D rises from 55.31 to 98.08 and mean caption length from 5.03 to 10.30 words, matching the dataset's more descriptive reference style; VizWiz-VQA accuracy rises from 73.95 to 75.80. General-domain performance declines only modestly, by 0.50 points on VQAv2 and 5% relative on COCO-Caps.
🔧 Specifications
- Base:
google/paligemma-3b-mix-224(bfloat16) - Adapter: LoRA, rank 8, alpha 16, dropout 0.05 · 11.3 M trainable params (0.385 %)
- Target modules:
q/k/v/o_proj,gate/up/down_proj - Data: 137,678 train / 27,569 val · seed 123
- Schedule: 3 epochs, 3,225 steps · effective batch 128 (16 × 8 grad-accum) · max seq 512
- Optimiser:
adamw_torch_fused· LR 2e-4, cosine (warmup 0.03) - Parameters: 2.92 B (adapter merged into the base weights)
- Precision: bf16 (merged, standalone)
- In-memory size: 5.45 GB (
model.get_memory_footprint()) - Training peak VRAM: 53.1 GB
📊 Measured Performance
Blind-domain (target task) — VizWiz
VizWiz-VQA
| Model | Overall | Yes/No | Number | Other | Unans. |
|---|---|---|---|---|---|
| Base | 73.95 | 88.71 | 66.10 | 66.10 | 91.67 |
| LoRA | 75.80 ±0.21 | 86.55 | 67.56 | 66.54 | 97.73 |
VizWiz-Caps
| Model | CIDEr-D | BLEU-4 | METEOR | ROUGE-L | Len. |
|---|---|---|---|---|---|
| Base | 55.31 | 12.07 | 14.08 | 28.76 | 5.03 |
| LoRA | 98.08 ±1.51 | 30.77 ±0.36 | 23.45 ±0.31 | 49.54 ±0.47 | 10.30 |
General-domain (control) — VQAv2 / COCO-Caps
VQAv2
| Model | Overall | Yes/No | Number | Other |
|---|---|---|---|---|
| Base | 81.65 | 94.74 | 67.53 | 73.49 |
| LoRA | 81.15 ±0.04 | 94.57 | 66.38 | 72.85 |
COCO-Caps
| Model | CIDEr-D | BLEU-4 | METEOR | ROUGE-L | Len. |
|---|---|---|---|---|---|
| Base | 131.21 | 31.96 | 30.62 | 59.17 | 12.40 |
| LoRA | 124.54 ±1.36 | 34.58 ±0.39 | 30.63 ±0.06 | 58.62 ±0.19 | 11.20 |
LoRA/QLoRA scores are mean ± std over 3 seeds (42, 123, 7). This checkpoint is the seed-123 run.
🚀 Usage
from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch
repo = "lamao-ab/paligemma-blind-assist-lora-merged-v1"
model = PaliGemmaForConditionalGeneration.from_pretrained(
repo, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = PaliGemmaProcessor.from_pretrained(repo)
image = Image.open("example.jpg").convert("RGB")
prompt = "describe the scene for a blind person"
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))
📄 License
Inherits the Gemma license from the base model.