SigLIP Skin Lesion Classifier

Fine-tuned SigLIP vision model for skin lesion classification and triage. Part of the SkinTag project for equitable melanoma detection.

Model Details

Base Model: google/siglip-so400m-patch14-384 (878M parameters)

Fine-tuning: Last 4 vision transformer layers unfrozen (~7% trainable parameters), with classification head. Trained with Fitzpatrick-balanced sampling for fairness across skin tones.

Training Data: 47,277 images from 5 dermatology datasets:

Performance

Binary Triage (Benign vs Malignant)

Clinical Thresholds:

• At 95% sensitivity: 82.5% specificity
• At 90% sensitivity: 88.3% specificity

Condition Estimation (10-class)

10 categories: Melanoma, BCC, SCC, Actinic Keratosis, Melanocytic Nevus, Seborrheic Keratosis, Dermatofibroma, Vascular Lesion, Non-Neoplastic, Other/Unknown.

Fairness

Tested across Fitzpatrick skin types I-VI with <5% sensitivity gap. See evaluation_results.json for per-group metrics.

Versioning

To load a specific version:

from huggingface_hub import snapshot_download
# Latest (main)
snapshot_download("skintaglabs/siglip-skin-lesion-classifier")
# Rollback to v1
snapshot_download("skintaglabs/siglip-skin-lesion-classifier", revision="v1-original")

Files

Root (Model Weights)

• config.json -- Architecture configuration (model_name, hidden_dim, n_classes, dropout, unfreeze_layers)
• head_state.pt -- Classification head weights (~1.2 MB)
• model_state.pt -- Full fine-tuned model weights (~3.5 GB)

Misc/ (Pipeline Artifacts)

• classifier.pkl, classifier_logistic.pkl, classifier_deep.pkl -- Binary classifiers (frozen embeddings)
• classifier_deep_mlp.pkl -- Alias for classifier_deep.pkl (web app compatibility)
• classifier_xgboost.pkl -- XGBoost binary classifier (frozen embeddings)
• classifier_xgboost_finetuned.pkl -- XGBoost on fine-tuned embeddings (AUC 0.960)
• xgboost_finetuned_binary.pkl, mlp_finetuned_binary.pkl -- Fine-tuned pipeline classifiers
• xgboost_finetuned_condition.pkl -- 10-class condition classifier (fine-tuned)
• classifier_condition.pkl, classifier_condition_deep.pkl, classifier_condition_logistic.pkl -- Condition classifiers
• embeddings.pt -- Frozen SigLIP embeddings (~109 MB)
• embeddings_finetuned_train.pt, embeddings_finetuned_test.pt -- Fine-tuned embeddings
• metadata.csv, test_metadata.csv -- Dataset metadata with labels
• evaluation_results.json, training_results.json -- Metrics

Usage

End-to-End Inference (Recommended)

from src.model.deep_classifier import EndToEndClassifier
model = EndToEndClassifier.load_for_inference("path/to/downloaded/model")
probs = model.predict_proba(images)  # Returns [benign_prob, malignant_prob]

Embedding + Classifier Head

from transformers import AutoModel, AutoImageProcessor
import pickle

# Extract embeddings with frozen SigLIP
processor = AutoImageProcessor.from_pretrained("google/siglip-so400m-patch14-384")
model = AutoModel.from_pretrained("google/siglip-so400m-patch14-384")
embeddings = model.vision_model(pixel_values=processed_images).pooler_output

# Classify with downloaded head
with open("Misc/classifier.pkl", "rb") as f:
    clf = pickle.load(f)
predictions = clf.predict_proba(embeddings.numpy())

Full Implementation

See the SkinTag repository for the complete pipeline including web app, mobile distillation, and evaluation.

Citation

@misc{skintag2026,
  title={SkinTag: AI-Powered Skin Lesion Triage for Equitable Melanoma Detection},
  author={SkinTag Labs},
  year={2026},
  url={https://github.com/skintaglabs/main}
}

Base model (SigLIP):

@misc{zhai2023sigmoid,
  title={Sigmoid Loss for Language Image Pre-Training},
  author={Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},
  year={2023},
  eprint={2303.15343},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

Attribution

• Base Model: SigLIP by Google (Apache 2.0)
• Fine-tuning: SkinTag Labs (MIT)

Medical Disclaimer

This model is for research and triage screening only, not clinical diagnosis. Users should consult healthcare professionals for medical advice.

License

MIT License (fine-tuned weights). Base model (SigLIP) is licensed under Apache 2.0 by Google.