pallet-pose-yolo26m-ft

Single-stage pallet 6-DoF pose model for forklift deployment. One RGB frame in, one bounding box + 9 keypoints out; a downstream PnP turns those keypoints into a 6-DoF pose (yaw / lateral offset / forward distance).

This is the medium variant (23.6M params). A smaller sibling trained on identical data with an identical recipe is at CanelE452/pallet-pose-yolo26n-ft (nano, 2.7M params) — see Which one should you use, because the medium is not clearly better.

⚠️ Read this before running inference

Two contract details are not optional. Get either wrong and the model looks broken.

1. Reflect-pad the image by 100 px on all four sides

The model was trained on padded frames so that pallets clipped by the image border stay learnable. Match that at inference:

padded = cv2.copyMakeBorder(img, 100, 100, 100, 100, cv2.BORDER_REFLECT_101)

Subtract 100 from predicted coordinates before doing anything geometric (PnP, drawing).

2. Keypoint order is camera-facing, not object-fixed

Index 0-3 is always the face pointing at the camera — it is not tied to a fixed side of the physical pallet.

0 near_top_left      1 near_top_right      <- face toward the camera (fork pockets)
2 near_bottom_right  3 near_bottom_left
4 far_top_left       5 far_top_right       <- opposite face
6 far_bottom_right   7 far_bottom_left
8 centroid

top    = {0, 1, 4, 5}
bottom = {2, 3, 6, 7}

Never enable horizontal flip augmentation if you finetune this further. The ordering is left-right asymmetric; fliplr silently corrupts it. Training used fliplr=0.0. If you do enable it, the correct mapping is flip_idx: [1, 0, 3, 2, 5, 4, 7, 6, 8] (0↔1, 2↔3, 4↔5, 6↔7, centroid fixed) — the identity permutation is wrong.

Usage

import cv2, numpy as np
from ultralytics import YOLO

model = YOLO("pallet_yolo26m_pose_ft.pt")
PAD = 100

img = cv2.imread("frame.png")                     # 640x480 BGR
padded = cv2.copyMakeBorder(img, PAD, PAD, PAD, PAD, cv2.BORDER_REFLECT_101)

r = model.predict(padded, imgsz=640, conf=0.4, verbose=False)[0]
if r.boxes is not None and len(r.boxes):
    i = int(np.argmax(r.boxes.conf.cpu().numpy()))     # highest-confidence instance
    kps = r.keypoints.xy.cpu().numpy()[i] - PAD        # (9, 2) in original image coords
    conf = float(r.boxes.conf.cpu().numpy()[i])

Recovering 6-DoF pose

Feed the keypoints to PnP with the pallet's known 3-D corner model. In the reference deployment: SOLVEPNP_SQPNP + refineLM, using only keypoints whose confidence ≥ 0.5. Index 8 (centroid) is a real correspondence and may be included.

Why the finetune

The synthetic-only pretrain had a structural blind spot: all 73,916 training images contained a pallet, so the model had never seen a pallet-free scene. On deployment video it confidently labelled the forklift's own forks and a metal fence as pallets — both are horizontal-slat structures that genuinely resemble a pallet.

The fix was not more augmentation or class weighting. It was adding background images (empty labels) from the deployment camera itself.

Results

Evaluated on a held-out real set of 161 manually annotated frames excluded from training, and on the 911-frame deployment sequence.

False positives — frames with no pallet (n=259)

Detection rate on pallet-free frames, swept over confidence threshold (lower is better):

These 259 frames were part of the finetune set, so this figure is in-sample. The held-out numbers below are the honest ones.

Held-out real evaluation set (n=161, never trained on)

Keypoint error is the median L2 distance over visible keypoints, in original-image pixels.

Which one should you use

Paired comparison on the 153 held-out frames both models detect:

medium more accurate   87 frames
nano   more accurate   66 frames
median difference   -0.24 px  (medium better)
mean   difference   +1.48 px  (medium worse — driven by its tail)
Wilcoxon p = 0.14   -> NOT statistically significant

Error-tail breakdown:

The medium is better in the middle of the distribution and worse in the tail. It wins more frames than it loses, but not by a statistically significant margin.

Inference latency (RTX 3080, 640×640 padded, single frame):

Recommendation: start with the nano — it is 8.7× smaller, has a tighter error tail, and the accuracy difference is not significant. Take the medium if your platform has room and you care about typical-case precision more than worst-case, or if you plan to finetune further on more data (the larger model has more headroom).

Known limitations

Both variants share these — they come from the data, not the architecture.

• Heavily clipped pallets can be missed. One deployment frame with a pallet cut off at the bottom edge went 0.427 (pretrain) → 0.000 (finetuned). Training longer does not fix it; this comes from the background-image composition, not from undertraining.
• Distant, small pallets are weaker for the same reason.
• One pallet per frame. Trained single_cls=True with one instance per image; the reference pipeline takes the highest-confidence box. Multi-pallet scenes are untested.
• Fixed camera geometry. Tuned for a 640×480 forklift-mounted camera. Other mounting heights and view angles are untested; real training data skews to low elevation angles.
• Not a general pallet detector. Plastic pallets dominate the real data. Wooden pallets were explicitly excluded.

Tuning conf

0.4 is the deployment default. Because the false-positive rate is 0% even at 0.05, you can lower the threshold to 0.25 to recover some distant/clipped detections at little risk.

Training

Synthetic images are kept in the mix so the finetune does not forget the keypoint structure learned during pretraining — 157 real images alone are 0.2% of the pretrain set.

patience=0 disables early stopping deliberately: with the default patience the nano run stopped at epoch 16 and saved epoch 1 as "best", because Ultralytics' fitness is computed on the synthetic validation split.

A note on validation metrics

The bundled validation set is synthetic, and its mAP does not predict real-world quality. On the nano variant, pushing synthetic pose mAP from 0.9448 to 0.9573 produced no measurable real improvement (paired Wilcoxon on the held-out set, p = 0.83). Judge changes on real data.

Files

License

AGPL-3.0, inherited from the Ultralytics YOLO base weights. If you deploy this over a network service, AGPL obligations apply to your application. For a commercial license, contact Ultralytics.