DermaTech Qwen 2 VL LoRA Model
This model is a fine-tuned version of Qwen/Qwen2-VL-2B-Instruct using Low-Rank Adaptation (LoRA) for specialized dermatological diagnosis. It's designed to analyze images of skin conditions and provide insightful diagnostic descriptions, mimicking the expertise of a dermatologist. This model is a key component of the DermaTech project (GitHub Repo).
Model Description
The base model, Qwen 2 VL, is a powerful multimodal model pre-trained on a massive dataset of text and image pairs. This LoRA adaptation focuses its capabilities on dermatology by training on a curated dataset of 1500 image-text pairs, where images depict various skin conditions and the text provides corresponding expert diagnoses. This targeted training allows for more accurate and relevant responses in the context of skincare analysis.
Training Procedure
- Base Model: Qwen/Qwen2-VL-2B-Instruct
- Training Dataset: A custom dataset of 1500 image-text pairs related to dermatological conditions (Dermatech_vlm.json).
- Fine-tuning Method: LoRA (Low-Rank Adaptation)
- Optimizer: LoRA+ with a learning rate ratio of 16.00.
- Learning Rate Schedule: Cosine Decay
- Batch Size: 2 per device, effective batch size of 8 with gradient accumulation of 4.
- Training Steps: 618
- Training Epochs: 4 (though training completed early due to cosine decay schedule)
- Hardware: Trained on a single GPU (cuda:0).
- Training Loss: 0.5477
- Training Time: Approximately 9 minutes.
The model was resumed from checkpoint-205 and saved at checkpoint-618.
Intended Uses & Limitations
This model is intended for research and prototyping purposes as part of the DermaTech project. It's designed to assist in preliminary skin condition assessment and provide information, but it should not be considered a substitute for professional medical advice. Always consult with a qualified dermatologist for accurate diagnoses and treatment plans.
The model's performance might be limited by the size and diversity of the training data. It may not accurately diagnose all skin conditions, and its output should be interpreted with caution.
How to Use
You can use this model with the transformers library. Refer to the DermaTech GitHub repository for detailed usage instructions and integration with the full application.
Evaluation Results
Formal evaluation is still pending, but initial results show promising performance on the training data. Further testing and benchmarking will be conducted to assess generalization and accuracy on unseen data.
Example Usage (Code Snippet - adjust as needed)
from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor, GenerationConfig
# Load model and processor
model = Qwen2VLForConditionalGeneration.from_pretrained("Rewatiramans/Dermatech-Qwen2-VL-2B")
processor = Qwen2VLProcessor.from_pretrained("Rewatiramans/Dermatech-Qwen2-VL-2B")
# Load and process image
...
# Prepare inputs
...
# Generate text
generation_config = GenerationConfig(do_sample=True, temperature=0.7) # Adjust parameters as needed
outputs = model.generate(**inputs, generation_config=generation_config)
# Decode output
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]
print(generated_text)
Citation
If you use this model in your research, please cite the DermaTech project and the original Qwen 2 VL model.
Contact
For any inquiries regarding the model or the DermaTech project, please contact [your_email_address].