PyraFuse: skin, fabric, and background segmentation
PyraFuse is a DINOv3-based semantic-segmentation framework for three classes:
0 = background, 1 = fabric, and 2 = skin. The associated paper has been
accepted at AIMLSystems 2026.
Available checkpoints
| Variant | Encoder | Folder |
|---|---|---|
small | DINOv3 ViT-S/16 | small/ |
small_plus | DINOv3 ViT-S+/16 | small_plus/ |
base | DINOv3 ViT-B/16 | base/ |
large | DINOv3 ViT-L/16 | large/ |
Each folder is self-contained: config.json, decoder.pt, ema.pt, and a
backbone/ directory with the DINOv3 configuration, model.safetensors, and
adapter feature-normalisation weights. Use EMA weights for evaluation.
Usage
Install the PyraFuse code repository, then load any variant. Pin a revision for reproducible experiments.
from pyrafuse import load_pretrained
model = load_pretrained("base", revision="v1.0.0", device="cuda")
Inputs must be RGB, ImageNet-normalised, and have height and width divisible by 16. The default checkpoint configuration uses 448 × 448 inputs.
Mobile and edge: ExecuTorch programs
The mobile/ folder contains ready-to-run ExecuTorch
programs for Android, iOS, React Native, Flutter, and embedded Linux. The same
.pte file runs in every framework that embeds the ExecuTorch runtime.
mobile/
├── manifest.json # sizes, SHA-256, delegation, held-out mIoU
├── small/pyrafuse_small_<target>.pte
├── small_plus/pyrafuse_small_plus_<target>.pte
└── base/pyrafuse_base_<target>.pte
| Target | Runs on | Precision |
|---|---|---|
xnnpack_fp32 | CPU: Android, iOS, macOS, Linux/ARM boards | FP32 |
xnnpack_int8 | same CPUs, about 4× smaller | INT8 weights, dynamic INT8 activations |
coreml_fp32 | iOS 17+ / macOS 14+ GPU and CPU via Core ML | FP32 |
vulkan_fp32 | Android GPU via Vulkan | FP32 |
| Variant | Program | Size | mIoU | Agreement with PyTorch |
|---|---|---|---|---|
small | PyTorch reference | 0.907 | ||
xnnpack_fp32 | 91 MB | 0.907 | 100.00% | |
xnnpack_int8 | 23 MB | 0.906 | 99.78% | |
coreml_fp32 | 92 MB | on device | on device | |
vulkan_fp32 | 91 MB | on device | on device | |
small_plus | PyTorch reference | 0.908 | ||
xnnpack_fp32 | 119 MB | 0.908 | 100.00% | |
xnnpack_int8 | 31 MB | 0.907 | 99.74% | |
coreml_fp32 | 121 MB | on device | on device | |
vulkan_fp32 | 119 MB | on device | on device | |
base | PyTorch reference | 0.912 | ||
xnnpack_fp32 | 348 MB | 0.912 | 100.00% | |
xnnpack_int8 | 88 MB | 0.912 | 99.80% | |
coreml_fp32 | 349 MB | on device | on device | |
vulkan_fp32 | 348 MB | on device | on device |
mIoU is measured on the first 200 images of the held-out test split at 448 px by executing each program with the ExecuTorch runtime; the Core ML and Vulkan programs need Apple or Android hardware and are not yet scored.
Contract. Input float32 [1, 3, 448, 448]: RGB, stretch-resized, divided by
255 (ImageNet normalisation is inside the program). Output float32 [1, 3, 448, 448]
logits for background, fabric, skin; take argmax over dim 1.
from executorch.runtime import Runtime
from pyrafuse import download_mobile_model # pip install "pyrafuse[mobile]"
forward = Runtime.get().load_program(
download_mobile_model("small", "xnnpack_int8", revision="v1.2.0")).load_method("forward")
logits = forward.execute([image])[0] # image: float32 [1, 3, 448, 448] in [0, 1]
React Native (react-native-executorch 0.10+) loads these files directly with
useSemanticSegmenter using labels ['background', 'fabric', 'skin'],
resizeMode: 'stretch', and normalizeOpts: { alpha: 1 / 255, beta: 0 }.
Copy-paste snippets for React Native, Flutter, Kotlin, Swift, and C++ are in the
code repository README.
There are no FP16 programs: DINOv3's first block produces attention logits up to about 2.5 × 10⁶, beyond the FP16 range, so FP16 backends (including the Core ML Neural Engine) output NaNs. Core ML and Vulkan programs are validated at export but have not yet been benchmarked on devices.
Release policy
These eager PyTorch checkpoint bundles are the portable source of truth; the
mobile/ ExecuTorch programs are exported from them with the repository's
scripts/export_mobile.py.
TensorRT engines are deliberately not distributed here because they are tied to
the target GPU, CUDA, TensorRT version, precision, and optimization profile.
Build and validate TensorRT engines on the deployment host from the PyTorch
checkpoint using the repository's scripts/export_trt.py command.
Licence
The PyraFuse decoder and repository code are Apache-2.0. Each checkpoint
bundle includes Meta DINOv3 backbone material; use and redistribution of those
bundles are additionally subject to the DINOv3 License. A verbatim copy is
provided at LICENSES/DINOv3-LICENSE.md in this model repository. The
underlying Fashionpedia and visuAAL datasets are not redistributed and retain
their own terms.
Data, limitations, and responsible use
Training labels fuse Fashionpedia annotations with visuAAL skin masks. The source data is not redistributed in this repository; users must comply with the source datasets' terms. Model performance can vary by image quality, lighting, occlusion, garment appearance, and representation in source data. This model is for research and engineering use, not a medical or biometric decision system.
Citation
The accepted manuscript is included in the code repository. Formal citation metadata will be added after the AIMLSystems 2026 camera-ready publication.