ChessSight corner heatmap (Swin-T)

Finds the four corners of a chessboard in a photograph. Those four points give the board-plane homography, which is what turns a pile of piece detections into an actual position.

0.128 squares mean corner error on the real ChessReD test split, with usable geometry found on 306 of 306 photographs — trained on synthetic renders only, never on a real photograph.

Licence — read this first

CC BY-NC 4.0. Commercial use is not permitted.

Trained on tchauffi/chesssight-synthetic-40k, about 60% of which renders the Staunton Chess Set by uppalong (Printables 76438), licensed CC BY-NC 4.0. Whether trained weights are a derivative work of their training data is legally unsettled; this model is labelled NonCommercial because its data is, rather than leaving you to discover the question later.

Usage

Needs only torch, timm, torchvision, numpy, pillow and safetensors — the project it came from is not required.

from huggingface_hub import snapshot_download
from PIL import Image
import sys

path = snapshot_download("tchauffi/chesssight-corner-heatmap-v1")
sys.path.insert(0, path)
from modeling_corner_heatmap import load_corner_model, predict_quad

model, config = load_corner_model(path, device="cuda")
corners = predict_quad(model, Image.open("board.jpg"), config, device="cuda")
# [[x, y], ...] four points, clockwise, or None if fewer than four were found

Then, for the homography:

import cv2, numpy as np
board = np.array([[0, 0], [8, 0], [8, 8], [0, 8]], dtype=np.float32)
homography, _ = cv2.findHomography(board, np.asarray(corners, dtype=np.float32))

How it works, and why

One heatmap channel, not four. Four separate channels would name the corners a8/h8/h1/a1, and that naming is not learnable from a board's appearance — rotate the photograph and the image is equally consistent with any assignment. So the model predicts a single "corner-ness" map with four peaks, and ordering is geometric. Which corner is a8 is a separate problem, solvable from square colour parity plus where White's men are (99.0% on ChessReD; see the repo).

Peak suppression, then sub-cell refinement. A 3×3 maximum filter keeps only cells that dominate their neighbourhood, which makes duplicate corners structurally impossible rather than something to filter afterwards — four points containing a duplicate give a degenerate homography rather than an obviously wrong one. A local intensity-weighted mean then places each point between cells, because a whole-cell answer is quantised to 4 input pixels, which a 3072px photograph scales up sevenfold.

Swin, not ViT. A patch-16 ViT's finest feature map is stride 16; upsampling it 4× to reach the heatmap grid discards exactly the localisation this design exists for. Swin's patch-4 stem gives stride 4 natively. Swin-T also removed the failure tail a ResNet-18 backbone had: p90 error fell from 323px to 55px.

Results

ChessReD test split, 306 photographs, against alternatives measured identically:

Used as the geometry stage of the full pipeline (with chesssight-rtdetr for pieces), the end-to-end position read is 97.06% of squares and 31.05% of boards exactly right.

Limitations

• Every corner must be in frame. A heatmap peak has to land on a pixel, so a cropped board cannot be solved. On tournament video where the near edge is cut off, geometry was found on 8 of 48 sampled frames; the board's bounding box was found on 48 of 48. If your boards are cropped, this is the wrong tool.
• It always returns four points. topk does not know what a corner is, so pass min_score to let it answer "no board here" — otherwise a photograph of a kitchen yields a quad. Peak confidence separates well out of domain (successes ≈0.5–0.6, failures ≈0.1–0.3), but a threshold fitted on clean data comes out at zero, because clean data contains no failures to separate.
• Scale bias of about 1.4%: predicted quads come back slightly large on real boards. Down from 3.3% before the training data varied its border tone, so most but not all of it was a generator artefact.
• Scale-specific. Trained at 448px; inference at 640 or 896 degrades monotonically. Unlike the detector, it gains nothing from a larger input.
• Synthetic training only. Unusual board styles — outdoor sets, ornate pieces, strongly coloured squares — are out of distribution and fail more often.

Training

Swin-T pyramid → stride-4 top-down path → one heatmap channel. CenterNet penalty-reduced focal loss (a stride-4 map has ~12,500 cells and four are positive, so plain BCE is 99.97% correct predicting nothing). 25 epochs, batch 32, AdamW at 3e-4 with the backbone at a tenth of that, cosine schedule.

Checkpoint selection is on the real ChessReD validation split, not on renders: synthetic corner error plateaus around 4px while real error is still moving, so selecting on renders past that point picks on noise in the wrong domain.

Code, configs and evaluation: https://github.com/tchauffi/ChessSight