SigLIP-SO400M-patch14-384 - Renesas X5H
Introduction
This repository contains the SigLIP-SO400M-patch14-384 dual-encoder model, optimized for the Renesas X5H platform for zero-shot image-text similarity inference.
SigLIP is a dual-encoder model consisting of a vision encoder and a text encoder. It is not a generative model -- it does not produce text output. Instead, it computes similarity scores between images and text labels, enabling zero-shot image classification and image-text matching.
- Model Architecture: SigLIP uses a Vision Transformer (ViT) for image encoding and a Transformer for text encoding. Both encoders produce L2-normalized embeddings that are compared via dot-product similarity with a learned scale and bias, followed by sigmoid activation.
- Model Summary:
Parameter SigLIP-SO400M-patch14-384 HIDDEN_SIZE 1152 INTERMEDIATE_SIZE 4304 NUM_HEADS 16 HEAD_DIM 72 NUM_LAYERS 27 (both encoders) PATCH_SIZE 14 IMAGE_SIZE 384 NUM_PATCHES 729 VOCAB_SIZE 32000 MAX_TEXT_SEQ_LEN 64 - Source Model: google/siglip-so400m-patch14-384
Performance
The following performance metrics were measured on the Renesas X5H board.
| Precision | Device | VE Latency (ms) | TE Latency (ms) | NPU DDR (MB) |
|---|---|---|---|---|
| FP16 | NPX6 | 396.0 | 43.0 | 1708.04 |
Prerequisites
To run the model, you need:
- Renesas X5H Board with SDK v4.32.0 or v4.34.0
- Hugging Face CLI: For downloading the model and installer.
Board Setup
Power Cycle
The X5H board must be power-cycled via USB serial before first use or after any NPU hang.
echo "POWER#OF" > /dev/ttyUSB2
sleep 3
echo "POWER#ON" > /dev/ttyUSB2
Wait for the board to boot (typically 30-60 seconds).
NPU Setup
Run setup_npu.sh exactly once after each power cycle. Do not run it multiple times without rebooting first.
bash ./setup_npu.sh npu0
This loads kernel modules and starts all 14 NPU firmware cores.
Important: FP16 and W4A16 runners cannot be used back-to-back on the same boot. Running W4A16 corrupts NPU internal state, causing subsequent FP16 runs to produce NaN. Power cycle the board when switching between variants.
Deployment
SigLIP-SO400M-patch14-384 (FP16)
Two versions are available:
- xOS-v4.34: VE encoding
- Download the installer siglip-runner-3.0.0-Linux.sh from Files and versions tab under fp16/binaries/rcar-x5hv1/xOS-v4.34/ (optimized) or fp16/binaries/rcar-x5hv1/xOS-v4.32/ (baseline).
- Copy the installer to the X5H board and run the installer.
bash ./siglip-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license - Expected directory structure on the X5H board.
siglip-runner ├── model │ └── mmproj-siglip-f16.gguf ├── firmwares ├── kernel_modules ├── scripts ├── test_data │ └── car-1.ppm ├── siglip-runner └── setup_npu.sh
Inference - SigLIP-SO400M-patch14-384 (FP16)
bash ./setup_npu.sh npu0
./siglip-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -s
Expected output:
similarity_score: 6.5136025660e-03 (0.0065136026)
SigLIP-SO400M-patch14-384 (W4A16)
- Download the installer siglip-w4a16-runner-3.0.0-Linux.sh from Files and versions tab under w4a16/binaries/rcar-x5hv1/xOS-v4.34/ folder.
- Copy the installer to the X5H board and run the installer.
bash ./siglip-w4a16-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license - Expected directory structure on the X5H board.
siglip-w4a16-runner ├── model │ └── mmproj-siglip-f16.gguf ├── graphs │ ├── siglip-ve-w4a16 │ └── siglip-te-w4a16 ├── firmwares ├── kernel_modules ├── scripts ├── test_data │ └── car-1.ppm ├── siglip-w4a16-runner └── setup_npu.sh
Inference - SigLIP-SO400M-patch14-384 (W4A16)
bash ./setup_npu.sh npu0
./siglip-w4a16-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -g graphs -s
Expected output:
similarity_score: 5.3571168333e-02 (0.0535711683)
CLI Reference
Usage: siglip-runner -m <gguf> (-i <image> | -L <image_list>) (-t <token_ids> | -f <prompts_file>) [options]
-m Path to SigLIP GGUF model
-i Path to image (JPEG, PNG, BMP, or PPM)
-L File with one image path per line (batch image mode)
-t Comma-separated token IDs (single prompt mode)
-f Prompts file for batch mode (label|token_ids per line)
-d NPU device path (default: /dev/snps/arcnet0/app0)
-s Print performance metrics
W4A16 adds:
-g Graph directory (contains siglip-ve-w4a16/ and siglip-te-w4a16/)
Token IDs are SentencePiece encoded. The runner pads to 64 tokens internally and places EOS (token ID 1) at position 63.