Ship direction — a U-Net that reads heading through heavy occlusion
Estimates the heading of a ship icon in an 80x80 top-down game frame where the ship may
be almost entirely hidden behind portraits, village-name text, or map markers, and
reconstructs the whole ship at that heading. 0 deg = up (north), clockwise.
The network labels every pixel as a ship part; a rigid geometric fit then turns those part masks into an angle, refusing any pose that would place the hull on open water.
The bottom two rows are the point: with a single part visible — only a stern (t082), only a bow (t083) — the geometry still pins the heading.
Two stages
frame -> SegNet (U-Net) -> per-pixel part labels -> part_pose (rigid fit) -> heading
The checkpoint is only half the system. SegNet outputs (6, 80, 80) class probabilities
(bg, bow, hull, stern, sail_l, sail_r) and contains no notion of an angle. part_pose
(plain NumPy/FFT, in estimator.py) searches all rotations for the single rigid pose that
best explains those masks, scoring three terms:
| term | meaning |
|---|---|
| + part overlap | predicted parts match the canonical layout (recall on labels) |
| + green coverage | footprint covers the actual visible ship pixels (recall on pixels) |
| - open-water penalty | footprint must not sit on plainly visible water (precision) |
The last term is what keeps reconstructions physically consistent: a ship may hide under an occluder, but not float on open water where it would have been seen.
Usage
import numpy as np
from PIL import Image
from huggingface_hub import hf_hub_download
from estimator import ShipHeading # this repo's estimator.py
REPO = "fortunatetumbleweed/ship-direction"
est = ShipHeading(
model_path=hf_hub_download(REPO, "model.safetensors"),
canonical_path=hf_hub_download(REPO, "canonical.npz"),
)
img = np.asarray(Image.open("frame.png").convert("RGB")) # uint8 (80,80,3), RGB
r = est(img)
r.heading # float degrees, 0 = up/north, clockwise
r.parts_seen # e.g. ['bow']
r.labels # (80,80) uint8 part ids
recon = est.reconstruct(img, r) # (80,80,3) uint8, whole ship painted in
Requires torch, numpy, pillow, safetensors.
Input contract
A wrong shape raises; these fail silently:
- RGB channel order (OpenCV gives BGR — convert first)
- 80x80 uint8, the raw frame; normalization (
x/255 - 0.5) happens inside - ship at the game's fixed scale, roughly centered
Results
On 21 labelled real frames:
| set | mean error | within 20 deg |
|---|---|---|
| clean (12) | 0.7 deg | 12/12 |
| occluded (9) | 6.1 deg | 9/9 |
Speed on an M-series CPU: ~60 ms init, then ~39 ms/frame (10 ms network + 29 ms pose fit). Defaults to CPU on purpose — for a single 80x80 frame, GPU transfer overhead exceeds the compute saved.
Training
20,000 synthetic frames: the clean ship rendered at a random heading with real game occluders composited on top. Labels come free by rotating the canonical part map, so nothing was hand-annotated — and only visible parts are labelled, teaching the network to segment what it can actually see. Selected by lowest heading error on the real frames, not by pixel accuracy. 473k parameters, 22 epochs.
Honest limitations
- The 9 real occluded frames are the only real labelled data, and their backgrounds and the portrait asset were harvested to build the training occluders. Expect some optimism; the true test is fresh frames.
- Fixed to one ship sprite at one scale. A different icon needs retraining and a new canonical map.
- The green-based masks (
ship_green,open_mask_from_crop) are tuned to this game's palette and will not transfer unchanged.
A note on the labels
One dataset frame (t083) was originally labelled 210 deg, which is physically impossible — it places the ship on open water. This model read 174 deg; a heading-regression CNN and a template matcher both answered ~211 deg, i.e. they made the same mistake as the bad label. The label was corrected to 174. A black-box model agreeing with a label is not evidence the label is right.
License and provenance
Code and weights are Apache-2.0. The canonical ship sprite in canonical.npz and the
validation frames are derived from a commercial game's artwork; those underlying assets are
not covered by this license and remain the property of their rights holder. Published for
research and educational use.
Source: https://github.com/fortunatetumbleweed-commits/ship_direction