kairos-door-rain

Binary image classifier: rain vs no_rain, for a camera looking out a dock door. Fine-tuned from facebook/convnext-tiny-224 (Apache-2.0). Part of Kairos, a weather plugin for Boston Dynamics Spot fleets (Knight Hacks IX hackathon build). It feeds one input of a GO / HOLD rule.

Read this first

• Not yet tested on a Spot door view. No such labeled data exists publicly. Every number below is on public photos.
• It does not transfer across sources. In-source calibration (deduplicated): 95.7% rain recall, 1.3% false alarms. On dashcam road photos, AUROC is 0.562: close to a coin flip. Afternoon rain recall is 3.9% (4/102).
• The dashcam numbers are exploratory. That set was seen during development and shaped our data choices, so it is not a clean held-out test.
• Caution-only. In Kairos the camera can add a HOLD but can never turn a weather HOLD into GO. NWS / Open-Meteo rain alone holds the robot. Unknown camera state also holds.

How to use

import { pipeline } from "@huggingface/transformers";
const clf = await pipeline("image-classification", "builtbyd3v/kairos-door-rain", { dtype: "q8" }); // or "fp32"
await clf("door.jpg"); // [{ label: "rain", score }, { label: "no_rain", score }]

Python: pipeline("image-classification", model="builtbyd3v/kairos-door-rain"). Hold when p(rain) >= 0.4447 for q8, or >= 0.4728 for fp32 (each picked on the calibration split, one image per call). Both are in eval_results.json under thresholds.

Results

95% CIs are Wilson score intervals. Threshold rule: the highest threshold that keeps calibration rain recall >= 95%, rounded down. Scored with onnxruntime CPU on the shipped ONNX files, one image per call (as in the browser).

Calibration split (in-source, never trained on, duplicates grouped so none cross splits): 1,033 images (116 rain).

Weather-Time road images, exploratory (seen during development): 2,600 labeled dashcam photos (595 rainy; 2,005 sunny or cloudy). Never trained on or used for a threshold.

Per period, q8, exploratory:

Why it fails, our reading: the training photos are scenic weather shots; the dashcam photos are through a windshield, where rain shows as wiper streaks and wet asphalt. A door camera is a third domain. The next step is labeled frames from the real door camera.

Changes from v2

A review found three evaluation flaws. All fixed here, then retrained:

• Duplicate photos crossed train and calibration (85 SHA-identical groups). prepare.py now groups images by 8x8 average hash before the split: 503 groups (1,439 files) merged. v2's 0.4% calibration false alarm rate was optimistic.
• q8 was scored in batches of 32, but deployment scores one image. Dynamic int8 quantization depends on the batch, so v2's q8 threshold moved from 0.7861 to 0.7666 when re-scored at batch 1. Now batch 1 everywhere.
• One threshold was used for both dtypes. Now there is one per dtype.

Training data and licenses

Split of the two training sources: grouped by 8x8 average hash, then 70 / 15 / 15 over groups (train / val / calibration), stratified by source class, seed 0. Val picks the best epoch; calibration picks the thresholds. Train: 4,946 (513 rain).

Training

training/train.py (reused from spot-wetfloor-convnext-tiny; only labels changed): 4 epochs (best: 3), lr 5e-5, batch 32, light crop/flip/color jitter. Apple M4 Pro (MPS), about 14 minutes. No paid compute. training/export_onnx.py: opset 17, max logit diff vs PyTorch 3e-6. training/quantize.py: dynamic uint8, MatMul/Gemm only (33.8 MB vs 111.4 MB). training/evaluate.py: everything in the tables. Raw numbers: eval_results.json.

Known limits

• Domain gap (above). Treat p(rain) as a weak hint until it is scored on real door frames.
• 8x8 average hash catches exact copies and light re-encodes, not crops or rotations.
• The baseline was trained on the same Kaggle source as ours, so only its dashcam numbers are a fair comparison.
• Night, glare, a dirty lens, and drizzle are untested.