Fine-Grained Visual Classification on Plant Leaf Diseases
Project Page: SelfSynthX.
Paper on arXiv: Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
This model is a fine-tuned multimodal foundation model based on LLaVA-1.5-7B-hf, optimized for detecting and explaining plant leaf diseases using the Plant disease dataset.
Key Details
- Base Model: LLaVA-1.5-7B
- Dataset: Healthy and diseased leaves across multiple plant species
- Innovation:
- Self-Synthesized Data: Extracts and describes disease-specific visual symptoms using the Information Bottleneck principle.
- Iterative Fine-Tuning: Uses reward model-free rejection sampling to improve classification accuracy and explanation quality.
- Intended Use: Identification of plant leaf diseases with human-verifiable symptom descriptions.
How to Use
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration
model_id = "YuchengShi/LLaVA-v1.5-7B-Plant-Leaf-Diseases-Detection"
model = LlavaForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
).to("cuda")
processor = AutoProcessor.from_pretrained(model_id)
conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": "What disease does this leaf have?"},
{"type": "image"},
],
},
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
image_file = "plant-disease/test1.png"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to("cuda", torch.float16)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))
Training & Evaluation
- Training: Fine-tuned using LoRA on PlantVillage with iterative rejection sampling.
- Evaluation: Demonstrates superior accuracy and robust, interpretable explanations compared to baseline models.
Citation
If you use this model, please cite:
@inproceedings{
shi2025enhancing,
title={Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data},
author={Yucheng Shi and Quanzheng Li and Jin Sun and Xiang Li and Ninghao Liu},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=lHbLpwbEyt}
}