ChessSight detector v0.2.0

An RT-DETR detector that finds a chessboard and the pieces on it. Trained entirely on synthetic Blender renders — it has never seen a real photograph during training. Paired with the repository's corner model, it reads 99.32% of squares and 71.24% of boards exactly on the ChessReD test split of 306 real photographs (v0.1.0: 99.25% / 68.30%); standalone detection is mAP 0.621 (mAP@50 0.879).

14 classes: six piece types × two colours, plus board and corner. The corner boxes are coarse hints — the shipped position pipeline still uses a dedicated corner heatmap model for geometry.

Source, dataset generator and training code: github.com/tchauffi/ChessSight (tag v0.2.0). Previous release: chesssight-rtdetr-v0.1.0.

What changed since v0.1.0

• Prior-bias head init (π = 0.01, the focal-loss trick). The freshly initialised classification head used to leave scores compressed under ~0.05; compression is roughly halved and the Platt calibration moved from scale 4.69, bias 20.24 to scale 2.95, bias 10.60.
• Piece-identity training data: the procedural queen now carries a coronet (she was previously the king's profile minus the cross, and queen↔king was the worst real-photo confusion), rook merlon counts vary 3–8, per-letter height jitter ±6%, and 15% of positions are openings (plies 4–24) so back ranks are crowded. Both queens gained ~0.1 AP on val.
• Orientation: an own-half pawn vote breaks 180° ties (25 of 45 flipped boards fixed, zero regressions) — this lives in the repository pipeline, not the weights.
• Harder framing: camera margin out to 2.6×, so distant boards are represented.
• Raw mAP is statistically flat against v0.1.0 (0.636 → 0.621, noise floor ~0.02), but its composition moved where board reading needs it; boards-exact is the number this release was built for.

Read this before you use it: the scores need calibrating

Raw confidences are still compressed (ranking is good; absolute values are low). calibration.json ships a Platt scaling fitted on held-out real photographs (ChessReD val). Apply it and the scores become usable probabilities:

calibrated = sigmoid(scale * logit(raw) + bias)
scale = 2.9502    bias = 10.5962

Two operating points, because detection and board-reading are different objectives:

For reading a board, recall is worth far more than precision: the pipeline keeps only the best detection per square, so a spare low-scoring box is usually harmless while a missing one always costs a square. The threshold belongs to this checkpoint's score distribution — re-sweep it if you swap detectors.

Usage (transformers only)

import json, torch
from huggingface_hub import hf_hub_download
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForObjectDetection

REPO = "tchauffi/chesssight-rtdetr-v0.2.0"
processor = AutoImageProcessor.from_pretrained(REPO)
model = AutoModelForObjectDetection.from_pretrained(REPO).eval()

calib = json.load(open(hf_hub_download(REPO, "calibration.json")))

image = Image.open("board.jpg").convert("RGB")
with torch.no_grad():
    outputs = model(**processor(images=image, return_tensors="pt"))

results = processor.post_process_object_detection(
    outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.0
)[0]

# Calibrate, then threshold.
raw = results["scores"].clamp(1e-6, 1 - 1e-6)
scores = torch.sigmoid(calib["scale"] * torch.logit(raw) + calib["bias"])
keep = scores >= calib["threshold"]

for score, label, box in zip(scores[keep], results["labels"][keep], results["boxes"][keep]):
    print(f"{model.config.id2label[int(label)]:14s} {float(score):.2f} {box.tolist()}")

pipeline("object-detection") also works, but it thresholds on the raw scores: pass threshold=0.0179 for the detection operating point (and note the confidences it reports are still the uncalibrated ones).

Video

Per-frame detection flickers. The repository ships a tracker (chesssight train video --smooth): on one clip, frame-to-frame churn dropped from 2.30 pieces to 0.61. If you write your own loop, do something equivalent — enter at a high threshold and survive at a lower one, vote the class over a track's history, and damp the box.

Results

Real photographs — ChessReD test, 306 images

Never used for training, checkpoint selection or calibration.

Synthetic — train7 validation split, 2050 renders

mAP 0.783 (mAP@50 0.934, small 0.738). Not comparable to v0.1.0's 0.839: train7 is deliberately harder — framing out to 2.6× leaves the board barely a quarter of the frame, and opening-heavy positions crowd the back ranks.

What it does not do

• Precise geometry. The corner class is a hint, not a homography. Position readout in the repository uses a dedicated corner-heatmap model.
• Small pieces. mAP 0.43 small against 0.98 large; distant boards and low camera angles degrade badly. A negative result worth knowing: training this data at 896 px input collapsed on real small objects (mAP-small 0.041) — 640² renders upsampled teach a blur real photographs don't have. Native high-resolution rendering is the untried lever.
• Out-of-domain footage. On small, blurred, near-edge-on boards, piece scores saturate on people and background and the board box can come back an order of magnitude too large. The repository's guards (board gating, class-agnostic NMS, a 32-piece cap) bound the damage; none of them make it correct.
• Calibration is domain-specific. Fitted on ChessReD-like photographs; re-fit with chesssight train calibrate for other domains.
• Single seed. Every number here is from one training run. Differences under about 0.02 mAP are not distinguishable from noise.

Training

train7: 20 000 Blender/Cycles renders at 640×640 (configs/train7.yaml in the repository). Every data change was promoted by a matched-seed 4800-image A/B against the previous recipe, one variable per arm. Positions 55% real Lichess games / 30% uniform random / 15% openings; two chess sets (baked Staunton OBJ and procedural lathe profiles, queen coronet, sampled merlons, per-letter height jitter); board frame pinned dark (sampled light borders measurably cost the piece detector while helping the corner model); HDRI lighting and backdrop on every image; chess clocks and distractors; camera azimuth 0–360°, elevation 8–75°, 24–85 mm, framing margin 0.95–2.6.

16 epochs, batch 12, AdamW, lr 1e-4 (backbone 1e-5), cosine schedule, classification loss weight 3.0, head prior-bias init 0.01, augmentation, EMA (0.9999). 148 min on one RTX 5070 Ti.

Intended use

Research and analysis on chess imagery. Not validated for officiating, rating, or any setting where a misread board carries a cost.

Licence

CC BY-NC 4.0. Commercial use is not permitted. The model is trained on synthetic renders of which about 60% depict the Staunton Chess Set by uppalong (Printables 76438), licensed CC BY-NC 4.0. Whether trained weights are a derivative work of their training data is legally unsettled; this model is labelled NonCommercial because its data is, rather than leaving you to discover the question later. Remaining assets are CC0 (Poly Haven HDRIs and textures) and public Lichess game dumps. A 40k-sample release of the generator's output is published as tchauffi/chesssight-synthetic-40k under the same terms.