HAYABUSA Vision V3 — Retrospective

Perception stack for an autonomous racing drone I built for the AI Grand Prix (sim qualifiers, summer 2026). Three models, one repo. This page is the retrospective: what it was for, the approaches I took, what worked, what didn't, and the numbers. Each folder has its own card with the exact recipe, the code as run, every eval JSON and the session reports.

Detector weights are fine-tuned from Ultralytics YOLO11 and are AGPL-3.0. Teacher and student are under the license in the header.

What it was for

• The race. Quadcopter through a sequence of square gates, fastest time wins. Gates: 2.7 m outer frame, 1.5 m hole, 0.26 m deep, all identical. Sensors: one 640×360 camera at 30 Hz (fx = fy = 320, tilted up 20°) and an IMU. Nothing else.
• The constraint that forced this. The VQ2 spec removed the simulator's position feed. My previous stack used that feed as the baseline for parallax ranging, so it was gone overnight. What was left was a YOLO box and range = 480 / box_height_px.
• The measured defect. Against a geometric solve on 300 hand-labelled gates, that range law was off by a median of 13.7 % at small scale and 25.3 % at mid scale. Cause: an axis-aligned box around a rotated square inflates. Unfixable within the method.
• The fix. Regress the four corners of the 1.5 m aperture and solve PnP (IPPE). Full 6-DoF gate pose, no odometry needed. Vision V3 was the program to make that reliable.
• The rule everything was built around: when the model claims a pose it must almost never be wrong. Abstaining costs a few frames of tracker coast; hallucinating costs the airframe. Every eval weights false confidence over missed coverage.

The approaches

1. Two-stage pipeline, recall then precision

Full-frame YOLO for discovery (tuned for recall, ~1.8 boxes/frame), then a crop-level pose network on the chosen candidate. Disambiguation belongs to the pose model and the tracker, not the detector. The pose network's output goes through a nine-reason validation ladder before anything reaches the controller: ≥ 3 corners with evidence, heatmap spread ≤ 6 px, no multimodal peaks, convex quad, reprojection ≤ 2 px, no trusted pose under 20 px aperture, range-law agreement, mask agreement, temporal jump and IMU rotation checks. The ladder is the authority, not the network's confidence.

2. Heatmaps with variable sigma, amodal targets, separate visibility

One 64×64 heatmap per corner in a fixed gravity-anchored order (TL, TR, BR, BL). Target sigma is 1 + 0.02 × aperture_h_px, clipped to [1, 6] px, plus 2 px for occluded or uncertain corners. Targets are amodal: an occluded or out-of-crop corner still gets its Gaussian at the true projected spot; visibility is its own supervised head. Decode = argmax + 3×3 quadratic sub-pixel refinement; RMS spread of mass above 0.3 × peak becomes the per-corner uncertainty the ladder gates on.

3. Render the training data

The sim exposes no label truth and pauses when unfocused, so I wrote a deterministic z-buffer quad renderer in numpy + OpenCV that draws the gate solid, floor, pillars, occluders, distractors and the course ribbon in the sim's look, with a degradation chain (motion blur, defocus, exposure, bloom, chroma bleed, noise, JPEG, aliasing). Labels are exact projections; the render buffers give per-corner visibility, visible fraction, per-side occlusion and aperture masks for free. sample = f(profile, seed), bytewise-replay tested, geometry/taxonomy/generator hashes in every manifest. Final corpus: 48,017 frames across 11 datasets (balanced ~44 k, plus 2.5–3 k each for occlusion, stacking, oblique, tiny-far, 33–102° view angles, and two negatives sets).

4. Treat real labels as noisy and small

A few hundred hand-labelled sim frames in a web labeler with per-corner flags (visible / inferred / unknown) and a per-gate "is there a defensible pose" verdict. Measured human repeat consistency by blind re-labelling: 0.07 px median, 4.4 px p90, so real corners are supervised at 1 px sigma. Benchmark runs write-locked in the labeler; 15 % hold-out fenced by run and by content hash. Real tier oversampled 15× and still ~10 % of each epoch.

5. Hard negatives from the gate itself

After round one hallucinated on gate paintwork, 18 % of positive crops became hard negatives: slivers cut along the gate's band and off-centre spills, supervised with pose_valid = 0 only. pose_valid = 1 was tightened to require the aperture framed inside the crop.

6. One knob per experiment

r3a → r3b → r3c, s1 → s2 → iso1, ft1 → ft2: each pair differs by exactly one change, and each failed run stayed on disk as evidence. Promotion was by a written rule applied criterion by criterion, never by the newest number.

7. Truth-masked distillation

The student copies the teacher only where human labels don't contradict it. Dense terms distilled only where dense ground truth is legitimate; pose_valid distilled only on ground-truth-positive crops. On the human's "no valid pose" gates, the human verdict stands alone.

What was great

• Corner precision roughly doubled. Teacher on real frames it never saw: 0.96 px median, 1.75 px p90, 96.2 % within 2 px, 99.1 % within 5 % of aperture height. The box-based crop model it replaced: 2.16 px median, 46.4 % within 2 px.
• Calibration was essentially perfect. Expected calibration error of the pose_valid head: 0.000. High-confidence false poses on 60 gate-free crops: 0.000. Abstention recall: 1.000.
• The hallucination fix worked in one round. Gate-texture crops with pose_valid > 0.8: 28.7 % (r1) → 3.3 % (r2) → 0.0 % (r3b). Ladder accepts on every negative family: 0.
• Synthetic data closed a blind spot nothing else could. The old detector's recall on tiny far gates was 2.8 %, small far 33 %, gate-through-gate 19 %. After retraining on 30 k rendered frames: 71 %, 91 %, 84 %. The real benchmark had no far gates, so this hole was invisible until the renderer existed.
• Real coverage nearly doubled from the purified real tier. Ladder acceptance of valid hold-out poses: 33.3 % (r2) → 62.5 % (r3b). Benchmark ladder accepts: 8/27 → 15/27.
• The teacher was fast enough to fly. 11.2 ms median forward on an RTX 3090 vs 25.9 ms for the model it replaced. The "slow accurate teacher" went live at twice the speed of its predecessor.
• The leakage gate caught a real leak before training: one run was in the pose validation set and the detector training set at the same time. Build aborted, run removed.
• The outcome line. The acceptance review after the sixth validation flight on the new stack read "trajectory failure with honest observation." Perception stopped being why the drone crashed.

What didn't work

• The student. Twice as fast (5.3 ms), corners as accurate as the teacher (1.03 px hold-out median vs the teacher's 1.19), and far more honest on poison gates (15 % vs 76 % false yes). But heatmap spread was 7.8 px against the ladder's 6 px gate, so the ladder accepted 0/27 benchmark poses and 4.2 % of hold-out. Three one-knob ablations showed the distillation weight was not the cause (2.0 → 0.5 moved spread 8.56 → 7.77 px) and the gate-centred negative crops were not the cause (removing them: 7.34 px, but poison honesty collapsed back to 67 %). The spread came from the second purified lineage itself; the undistilled r3c on the same data had 8.10 px. Narrowed suspects: the 85 new audit positives and the fractional positive-replication scheme. Not resolved.
• The raw abstention head over-trusts visible-but-invalid gates. On 33 hold-out crops placed squarely on a gate the reviewer ruled has no defensible pose, r3b says pose_valid > 0.8 75.8 % of the time. The ladder contains this to 2/33. r3c fixed the head (6.1 %) but collapsed coverage to 12.5 %, so it was rejected. Never use the raw head alone.
• Over-weighting real data, twice. r3a at 40× real oversample with a 71 % negative real tier went timid and lost precision. Detector ft1 at ~25 % real per epoch recovered point-blank gates but gave back 11–15 points of occlusion recall on held-out synthetic. Both times ~9–10 % real per epoch at a 50/50 pos/neg balance was the fix.
• 39.3 % of the "safe" positive pool was poison. The 140-item positive audit found 47 visible-gate-no-pose, 7 that failed geometry at 3–6 px reprojection, 1 hallucination. Every model before r3 had trained on some of it.
• Benchmark is thin. 27 pose crops from one run, medium and near range dominated. No far-field or heavy-occlusion pose crops on real frames. Everything above is measured on that.
• Sim only. Every frame came from the AI Grand Prix simulator or a renderer matched to it. Sim-to-real was never tested. One gate design, one colour.

What I think was genuinely clever

• Labels that never touch pixels. The renderer's labels are projections of the canonical geometry; pixels only supply visibility truth. No label noise from the render pipeline, ever.
• Supervising humans at human precision. Measuring repeat consistency and setting real-label sigma from it, instead of pretending a mouse click is exact.
• Hard negatives from healthy gates. The hallucination was on gate texture, so the negatives came from gate texture. One round, 28.7 % → 3.3 %.
• Truth-masked distillation under a known-overconfident teacher. The student came out more honest than what it was distilled from (15 % vs 76 % on poison), and the ablation proved which mechanism did it.
• Honest overlays. Colour only where the model claims; a grey ABSTAINED banner otherwise. A flagged-invalid decode can never be mistaken for a prediction in review.
• Pinning the live model by sha256 in the flight adapter. When the runtime was patched, the pin correctly refused a mismatched stack mid-campaign.

Numbers at a glance

Teacher r3b, frozen real benchmark (27 pose crops / 60 gate-free): corner err 0.963 / 1.753 px (med / p90) · PCK@2px 0.962 · PCK@5%h 0.991 · vis acc 0.944 · abstention 1.000 · hc-false 0.000 · ECE 0.000 · ladder accepts 15/27 · PnP reproj 0.546 px. Hold-out (24 pos / 34 neg): corner err 1.190 px · ladder accept 62.5 % · poison pv > 0.8: 75.8 % · poison surviving ladder: 6.1 %. Runtime (3090): 11.23 / 16.77 ms forward, 313 MB VRAM.

Student s2, same benchmark: corner err 1.034 / 2.499 px · PCK@2px 0.830 · abstention 0.983 · hc-false 0.000 · ECE 0.022 · ladder accepts 0/27. Hold-out: corner err 1.025 px · accept 4.2 % · poison pv > 0.8: 15.2 % · poison surviving ladder: 3.0 %. Runtime: 5.27 / 7.27 ms, 113 MB VRAM.

Detector ft2, real benchmark (58 frames, conf 0.30): recall 1.000 in every size bucket (160+ px 9/9, 80–160 20/20, 40–80 45/45, 20–40 24/24) · FP/empty frame 0.05 · 1.81 preds/frame · mAP50 0.991 · mAP50-95 0.781. Held-out synthetic: tiny_far 0.627 · small_far 0.854 · medium 0.971 · near 0.951 · severe_occlusion 0.731 · gate_through_gate 0.807.

Training cost: teacher r3b 1,805 s, student s2 1,586 s, detector ft2 ~2,300 s, all on one RTX 3090.

Using the models

python teacher/infer.py --ckpt teacher/teacher_v3_r3b.pt --image gate_crop.jpg
python student/infer.py --ckpt student/student_v3_s2.pt --image gate_crop.jpg

from ultralytics import YOLO
boxes = YOLO("detector/gate_detect_v2_yolo11s_ft2.pt").predict(
    frame_bgr, imgsz=640, conf=0.30, half=True, verbose=False)[0].boxes

Crop each candidate to a square ≈ 1.9 × box height, resize to 256×256, feed BGR/255 with no mean-std normalisation, decode with code/heatmaps.py, gate with code/validation.py, solve against code/geometry.py. Per-folder cards have the full recipe and file list.

Hashes: teacher 38fddadcad2c86366a172e946c7c00e5694aee4d789f0f323d191b6cd7c680ca · student 57286a7f4f9c1f085ba8dff66f618de56f227c6468b49b0ec8ab03c52f0580c9 · detector 3f1c137d2ee3396a50d02f7e96ef7e7f5ab4a83dfd0afb5524f1d3e8959b9f0d. Built July 2026.