UrbanOmniDetect-2 is a single hybrid detection and pose network that detects all 80 COCO classes and, for every road user, regresses the eight projected corners of its 3D box. It runs on ego-vehicle, infrastructure, and aerial footage without camera calibration, and the same forward pass drives tracking, a bird's-eye view, and an offline refinement stage that turns per-frame detections into rigid, physically consistent trajectories.
The inference pipeline, training code, and documentation live in the GitHub repository. This repository hosts the released checkpoints.
Highlights
- Detection and 3D cuboids from one forward pass. A YOLO26 network with a Pose26 head is a full COCO-80 detector and also predicts cuboids for person, bicycle, car, motorcycle, bus, and truck. No auxiliary detector is needed.
- Calibration-free on any viewpoint. No camera intrinsics, depth estimation, or ground-plane priors are required at inference time.
- Trained on a mixture of 2D and 3D data. COCO and VisDrone (boxes only) ground the detector. Real keypoint data (KITTI, DAIR-V2X), CDrone, and rendered vehicles (MeshFleet, Objaverse) teach the cuboids through a masked pose loss, so box-only images never push the keypoint head.
- Five scales. From 2.6M to 57.6M parameters, all trained at 640 px. Running them at 960 px on aerial footage recovers far, small traffic while the cuboids stay tight.
- Fast with TensorRT. The X model runs in 4.6 ms per frame at 960 px in FP16 on an RTX PRO 6000, against 7.7 ms in PyTorch FP32, with cuboid corners agreeing to a median 0.9 px.
Checkpoints
Every checkpoint is the epoch of its run with the highest road-class pose mAP, followed by a class-branch-only fine-tune on real box data (COCO + VisDrone) that corrects class boundaries learned from renders. The pose head is untouched by that step. Metrics are on the mixed validation set (COCO, VisDrone, KITTI and DAIR-V2X, CDrone, renders) at 640 px.
| Model | Params (M) | COCO AP | VisDrone AP @960 | KITTI 2D AP (Mod.) | KITTI AP3D E / M / H | KITTI APBEV E / M / H | Checkpoint |
|---|---|---|---|---|---|---|---|
| N | 2.6 | 14.3 | 14.6 | 91.5 | 31.5 / 22.2 / 18.8 | 37.0 / 26.5 / 22.4 | urbanomnidetect_v2_n_640.pt |
| S | 9.9 | 21.3 | 19.1 | 94.6 | 42.9 / 32.5 / 27.6 | 49.7 / 37.5 / 31.0 | urbanomnidetect_v2_s_640.pt |
| M | 21.3 | 28.1 | 25.1 | 95.4 | 46.3 / 34.8 / 30.7 | 50.7 / 39.6 / 35.3 | urbanomnidetect_v2_m_640.pt |
| L | 25.7 | 28.7 | 25.1 | 96.5 | 53.6 / 39.4 / 33.1 | 56.7 / 42.5 / 36.9 | urbanomnidetect_v2_l_640.pt |
| X | 57.6 | 30.4 | 26.7 | 96.6 | 47.5 / 37.1 / 32.9 | 52.1 / 41.1 / 36.4 | urbanomnidetect_v2_x_640.pt |
Quick Start
Clone the code and download a checkpoint:
git clone https://github.com/mkturkcan/urbanomnidetect.git
cd urbanomnidetect
pip install -r requirements.txt # ultralytics 8.4.61 (tested), torch, opencv, scipy
huggingface-cli download mehmetkeremturkcan/UrbanOmniDetect-2 \
checkpoints/urbanomnidetect_v2_x_640.pt --local-dir .
The checkpoints unpickle a class defined in v2/hybrid_v2.py, so run from the repository root (or put it on PYTHONPATH) and import it before loading:
import v2.hybrid_v2 # registers HybridPoseModel26 for unpickling
from ultralytics import YOLO
model = YOLO("checkpoints/urbanomnidetect_v2_x_640.pt")
r = model.predict("frame.jpg", imgsz=960, conf=0.1, device="cuda:0")[0]
boxes, classes = r.boxes.xyxy, r.boxes.cls # COCO-80 class ids
kpts = r.keypoints.data # (N, 8, 3): x, y, cuboid confidence
Video with tracking, the bird's-eye view, and offline refinement:
python bev_realtime_v2.py --input clip.mp4 --kp-model checkpoints/urbanomnidetect_v2_x_640.pt \
--kp-imgsz 960 --kp-conf 0.1 --kp-vis 0.5 --class-conf 0:0.4,1:0.5,3:0.5 \
--suppress-nested 0.85 --smooth 11 --scale-lock --fixed-camera auto --bev-fit --refine \
--layout dashboard --device cuda:0 --output out.mp4
Add --export tensorrt --half to build an FP16 engine on first use (TensorRT 10.x). See USAGE.md for the full command reference.
Relation to UrbanOmniDetect
UrbanOmniDetect-2 builds on the calibration-free, view-agnostic formulation introduced in UrbanOmniDetect (CVPR 2026 DriveX workshop), but it is a separate model with different outputs:
| UrbanOmniDetect | UrbanOmniDetect-2 | |
|---|---|---|
| Network | Pose-only keypoint model (YOLOv8 to YOLO12) | Hybrid detection and pose network (YOLO26, Pose26 head) |
| Classes | car, person, bike | All 80 COCO classes, cuboids for 6 road-user classes |
| Training data | UrbanOmniView | COCO, VisDrone, KITTI, DAIR-V2X, CDrone, MeshFleet, Objaverse |
| Keypoint order | Indices 0 to 3 are top corners | Indices 0 to 3 are ground-contact corners |
| Loading | Stock Ultralytics | Requires v2/hybrid_v2.py from the GitHub repository |
The two sets of checkpoints are not interchangeable. The UrbanOmniDetect checkpoints remain available at mehmetkeremturkcan/UrbanOmniDetect.
Citation
If you use UrbanOmniDetect-2, please cite:
@software{turkcan2026urbanomnidetect2,
author = {Turkcan, Mehmet Kerem},
title = {{UrbanOmniDetect-2}: One Network for Detection and Calibration-Free {3D} Cuboids from Any Viewpoint},
year = {2026},
month = {9},
url = {https://huggingface.co/mehmetkeremturkcan/UrbanOmniDetect-2}
}
Acknowledgements
This work was supported by the NSF Engineering Research Center for Smart Streetscapes under Award EEC-2133516, NSF Grants CNS-2450567 and CNS-2038984, and by computing resources from the NVIDIA Academic Grant Program and the Empire AI Consortium.
License
Released under the GNU Affero General Public License v3.0.