PlantDetect V2

Built with DINOv3. This is the completed V2 mean-pooling model for 2,991 plant species from Western Australia. Supply one photo, or 2–4 distinct photos from the same plant observation. This release uses the epoch-12 EMA weights from the expanded-data BF16 training run. The CLI rebuilds the encoder (bf16) and classifier (fp32) from the code-free FP32 model.safetensors through the bundled src/models/ code, so the release ships no TorchScript graph. Optimizer state and training data are excluded.

Custom Kernels — ExecuTorch & FP8 Export

Native ExecuTorch W8A8 Inference

executorch/PlantDetect-WA-V2-INT8.pte is a W8A8 export of this mean-pooling checkpoint for CPU inference without PyTorch. It uses custom plant_quant::* kernels (INT8 weights with per-output-channel FP32 scales, FP16 activations, FP32 accumulation) and ships for the Arm I8MM, SDOT and plain FP16-FML paths. The PTE is single-photo (one image per encode); multi-photo fusion uses the fp32 classifier head with one embedding per photo.

• Samsung Galaxy S24 (Exynos 2400): ~2.2 s per 448x448 encode at 5 threads cold, 2.5-2.8 s sustained
• Allwinner A733 (Cortex-A76 + A55): ~19 s per encode at 2 threads on a cold board, 25-31 s once it throttles
• Accuracy: 86.11% top-1 on the separate WA single-photo benchmark (0.22 points below the FP32 export)

Running the PTE

It does not load on a stock ExecuTorch runtime — it requires the four custom plant_quant::* operators. The operator sources, build instructions and prebuilt runners for Windows x64, Linux x86-64 and Linux aarch64 are included:

executorch/windows-x64/plant_runner.exe executorch/PlantDetect-WA-V2-INT8.pte encode image.bin embedding.bin
executorch/windows-x64/plant_runner.exe executorch/PlantDetect-WA-V2-INT8.pte classify slots.bin out 4

executorch/linux-x64/plant_runner     executorch/PlantDetect-WA-V2-INT8.pte encode image.bin embedding.bin
executorch/linux-aarch64/plant_runner executorch/PlantDetect-WA-V2-INT8.pte encode image.bin embedding.bin

image.bin is float32 ImageNet-normalised RGB 448x448 (3*448*448 floats); slots.bin is 4*512 floats. Set OMP_NUM_THREADS (5 on an Exynos 2400, 2 on an Allwinner A733). Required ExecuTorch version and commit, ISA tiers (PLANTDETECT_MARCH), activation modes (PLANTDETECT_ACT) and profiling switches are documented in extension/moe/README.md. The aarch64 runner targets the armv8.2-a+dotprod+fp16 tier (Cortex-A76 and newer); rebuild it for older cores or for I8MM with extension/moe/build_linux.sh. The Windows runner uses the portable scalar kernels, because MSVC defines neither __aarch64__ nor the ACLE feature macros the NEON paths need; use the Linux aarch64 runner for the fast kernels. The downloadable Linux runners may need chmod +x first. The aarch64 runner is built on the target board: a build host with a newer glibc/libstdc++ yields a binary that will not start on older ones — the previous aarch64 runner required GLIBC_2.38 / GLIBCXX_3.4.32 and could not run at all on a Debian-based A733.

FP8 AdaRound Weights

fp8/model.safetensors is the AdaRound E4M3FN per-channel weight-only export of the same checkpoint. Adaptive rounding improved 107 of 108 layer reconstructions (0.875% aggregate MSE reduction over nearest rounding); see fp8/adaround_report.json.

Measured accuracy

The external benchmark was selected before predictions and audited for known training overlap. It samples species across three training-count bands and covers 271 of 2,991 classes. It is not a natural-use-frequency estimate or an all-class accuracy measurement. Its single-photo metric is directly comparable with the internal individual photos row, not the one-view fused cohort. Photos mostly come through the GBIF/iNaturalist source family; community labels and incomplete historical upstream observation IDs limit the audit. See evaluation.json for the benchmark manifest checksum and breakdown. These rows were measured from the source EMA checkpoint with GPU BF16 autocast. The FP32 export contains exactly the same weights; CPU FP32 predictions may differ on close calls.

Harness measurements

Every figure here was measured through the artifacts in this repository on the external WA single-photo benchmark (1,822 photos, 271 species), and the centre-crop baseline reproduces the released 86.33% exactly, so the deltas are attributable to the harness.

Calibration (on by default, no extra compute). Raw softmax was over-confident: mean top-1 confidence 0.962 at 0.863 accuracy, ECE 0.101. Dividing the logits by a fitted temperature of 1.90 brings mean confidence to 0.84 and cuts ECE to 0.040 in sample and 0.062 held out (grouped 5-fold cross-validation by species chose 1.904 +/- 0.014, so the scalar is not overfit). Temperature is a strictly monotone rescale, so top-1, top-5 and every argmax-based check are provably unchanged; this was checked explicitly across T in 0.25-12 with zero top-1 flips.

Test-time augmentation (--tta, off by default). Training used random-resized-crop at scale 0.5-1.0 from a 576px cache, but inference only ever saw the single centre 448 crop, so the model is crop-robust in a way the default harness did not exploit. --tta fuses 5, 10 or 12 crops of one photo. Fusion is in probability space; averaging the embeddings of crops of one photo measured worse than the plain centre crop (86.22% vs 86.33%), because crops of a single photo do not lie on the multi-view manifold the pooling was trained for.

Paired McNemar against the baseline at --tta 12: 26 photos fixed, 13 broken, net +13, p = 0.053. The top-5 gain is the more reliable of the two, and top-5 is also what feeds the candidate list and the taxonomic corroboration. TTA applies to a single photo only: with 2-4 real photos the multi-view path already pools better than any TTA of one photo, so the CLI rejects the combination rather than quietly ignoring it.

Clutter check (with --tta, advisory). A mixed frame - your plant standing in a bush of other species - is where the model is most likely to return a neighbour. Measured on 600 benchmark photos tiled with three other species, the number of distinct genera among one photo's crops is 1.14 for a lone plant, 1.10 for four photos of the same species, and 3.06 for a mixed frame (AUROC 0.947). At the shipped threshold of 3 it nudges 3.0% of single-plant photos and 1.7% of same-species frames, while catching 67.7% of the mixed frames the model gets wrong. It is advisory: it prints a warning and never vetoes an identification. Do not read it as a guarantee - on hand-picked cases it both false-alarmed on a same-genus mix (the model's own sub-crop errors invent spurious genera) and missed a four-genus frame that was still named at 0.82 confidence. The reliable fix is the one the warning asks for: crop to a single subject. It needs --tta > 1, since it reuses those crops; on the default path the JSON reports "checked": false.

TTA is opt-in because it costs one encoder pass per crop. On the machine used for these measurements a CPU 448x448 encode takes ~2.45 s at 4 threads, so one photo costs ~2.5 s by default and ~29 s at --tta 12; a CUDA device is far faster. The --tta 12 fused distribution is already well calibrated, so its scores are not scaled again.

Run locally

Requires Python 3.11+, PyTorch, timm, safetensors, Pillow, and NumPy. The CLI works on Windows and Linux; timm constructs the DINOv3 backbone and safetensors loads the bundled weights, so no separate project checkout or model download is needed.

pip install -r requirements.txt
python plant_v2_cli.py identify leaf.jpg flower.jpg --device cpu --topk 5

The CLI also accepts --images leaf.jpg flower.jpg, and info prints model metadata. Inputs are RGB, bilinear resized to 576 x 576, centre cropped to 448 x 448, normalized with ImageNet mean/std, and EXIF orientation is applied (phone photos routinely carry an orientation tag; pass --no-exif to disable). The encoder runs as bf16 on both CPU and GPU (the classifier stays fp32); bf16 is ~35x faster than fp32 for this model on CPUs with AVX512-BF16 (oneDNN: 1124 vs 32 GFLOP/s) and matches the bf16 autocast the accuracy numbers above were measured with.

Output is a short human-readable report by default; --format json emits the full structured result (strict JSON, so non-finite values become null). Exit codes make it scriptable: 0 identified, 1 not confidently known, 2 usage or input error (missing or corrupt image, checksum mismatch, unavailable device). Scores are calibrated: the softmax is divided by a fitted temperature (1.90), cross-validated on the external benchmark, which cuts expected calibration error from 0.101 to 0.062. --no-calibration gives raw softmax. On a single photo, --tta 5|10|12 trades encoder time for accuracy and adds an advisory clutter warning (see Harness measurements); it is rejected with 2-4 photos, which already pool better.

Unknown species

The classifier can only ever return one of the known taxa, so for a plant outside the taxonomy the honest answer is "not confidently a known species". The CLI requires several independent signals before it names a species, and otherwise reports rejected_unknown with the failing checks - it never silently snaps a guess to the nearest class.

• Taxonomic agreement (always available). The species head's pick must sit in the same genus and family that the genus and family heads predict, and with 2-4 photos every photo's genus must match the fused pick. This is the strongest signal and needs no extra download.
• Nearest-centroid distance (optional). Each known species has an embedding centroid built from training photos; the nearest-centroid cosine must exceed a threshold calibrated on held-out photos of known species, so the false-rejection rate on knowns is a chosen quantile rather than a guess.

The centroid bundle (species_centroids.npz + centroid_report.json, ~5.4 MB) is optional. Without it the CLI still runs and still applies the taxonomic checks, and says so in the output. It is never downloaded implicitly: when it is missing and you are at a terminal the CLI offers to fetch it, or you can run it explicitly.

python plant_v2_cli.py fetch-centroids          # cache it, sha256-verified
python plant_v2_cli.py fetch-centroids --force  # re-download

The cache defaults to tempfile.gettempdir()/plantdetect; override with --cache-dir or PLANTDETECT_CACHE_DIR. --yes skips the prompt, --no-download never offers.

--relaxed keeps only the calibrated distance gate (the taxonomic checks become advisory); --no-reject disables rejection entirely (unsafe - a species is always named); --min-centroid-cosine moves the threshold.

Measured both ways with a grouped 5-fold cross-validation (threshold chosen on 4 folds, scored on the held-out fold), against 1,822 known photos (external WA benchmark, 271 species) and 6,318 photos of 819 species the model was never trained on (image_expansion rows absent from the taxonomy):

Cross-validated: 72.6% +/- 4.8 known accepted and 83.8% +/- 0.8 unknown rejected at threshold 0.842 +/- 0.004 - essentially the same as in-sample, so the operating point is not overfit.

The distance gate cannot be both protective and usable: reaching ~95% unknown rejection costs ~54% of known species. Two earlier figures were wrong and are corrected here - a knowns-only calibration put the threshold at 0.26 (52% unknown rejection), and a 4-species unknown set (103 of 110 photos one seagrass) suggested 93.6% at 0.60. Neither survives contact with 819 real species. centroid_report.json records the full protection curve and the selection rule; --min-centroid-cosine moves the line. The always-on taxonomic checks reject only ~31% of unknowns on their own, so they corroborate rather than carry.

Attribution and terms

Built with DINOv3 (vit_base_patch16_dinov3.lvd1689m). DINO-derived weights remain subject to the included DINOv3 agreement. Project-specific code is Apache-2.0. No training or benchmark photos are redistributed. See NOTICE.md.