ChessSight detector v0.1.0

Licence — updated

CC BY-NC 4.0. Commercial use is not permitted. This model was previously labelled MIT; that was an oversight.

It is trained on synthetic renders of which about 60% depict the Staunton Chess Set by uppalong (Printables 76438), licensed CC BY-NC 4.0. Whether trained weights are a derivative work of their training data is legally unsettled; this model is labelled NonCommercial because its data is, rather than leaving you to discover the question later. The training data is published as tchauffi/chesssight-synthetic-40k under the same terms.

An RT-DETR detector that finds a chessboard and the pieces on it. Trained entirely on synthetic Blender renders — it has never seen a real photograph during training. On the ChessReD test split of 306 real photographs it reaches mAP 0.636 (mAP@50 0.884).

13 classes: six piece types × two colours, plus board.

Source, dataset generator and training code: github.com/tchauffi/ChessSight (tag v0.1.0).

Read this before you use it: the scores need calibrating

Raw confidences from this model top out around 0.05. A short RT-DETR fine-tune ranks boxes well while barely moving the classification logits off their prior, so the usual threshold=0.5 returns nothing at all and the model looks broken. It is not — the ranking is fine, the numbers are compressed.

calibration.json ships a Platt scaling fitted on held-out real photographs (ChessReD val). Apply it and the scores become usable probabilities:

calibrated = sigmoid(scale * logit(raw) + bias)
scale = 4.6947    bias = 20.2386    operating threshold = 0.3948

At that threshold: precision 0.829, recall 0.947, F1 0.884. The transform is monotone, so it cannot change mAP — it only makes the numbers mean something.

Usage (transformers only)

import json, torch
from huggingface_hub import hf_hub_download
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForObjectDetection

REPO = "tchauffi/chesssight-rtdetr-v0.1.0"
processor = AutoImageProcessor.from_pretrained(REPO)
model = AutoModelForObjectDetection.from_pretrained(REPO).eval()

calib = json.load(open(hf_hub_download(REPO, "calibration.json")))

image = Image.open("board.jpg").convert("RGB")
with torch.no_grad():
    outputs = model(**processor(images=image, return_tensors="pt"))

results = processor.post_process_object_detection(
    outputs, target_sizes=torch.tensor([image.size[::-1]]), threshold=0.0
)[0]

# Calibrate, then threshold.
raw = results["scores"].clamp(1e-6, 1 - 1e-6)
scores = torch.sigmoid(calib["scale"] * torch.logit(raw) + calib["bias"])
keep = scores >= calib["threshold"]

for score, label, box in zip(scores[keep], results["labels"][keep], results["boxes"][keep]):
    print(f"{model.config.id2label[int(label)]:14s} {float(score):.2f} {box.tolist()}")

pipeline("object-detection") also works, but it thresholds on the raw scores. The calibrated 0.3948 threshold corresponds to a raw score of 0.0121, so pass threshold=0.0121 for the same operating point — and note the confidences it reports are still the uncalibrated ones.

Video

Per-frame detection flickers badly. The repository ships a tracker (chesssight train video --smooth) that associates detections across frames: measured on one clip, frame-to-frame churn dropped from 2.30 pieces to 0.61, and the worst single-frame jump from 27 to 9. If you write your own loop, do something equivalent — enter at a high threshold and survive at a lower one, vote the class over a track's history, and damp the box.

Results

Real photographs — ChessReD test, 306 images

Never used for training, checkpoint selection or calibration.

The last checkpoint scores 0.634, so the result does not depend on which epoch is picked.

Synthetic — held-out test split, 1966 renders

mAP 0.839, mAP@50 0.943, mAP small 0.758.

The 0.839 synthetic vs 0.636 real gap is the sim-to-real gap, and most of it is in small objects (0.758 vs 0.466). Distant or heavily foreshortened pieces are the weakness.

What it does not do

• No position readout. It emits boxes, not board corners, so there is no homography and no FEN. Reconstructing a position needs corner regression, which this release does not have.
• Small pieces. mAP 0.47 small against 0.92 large; low angles and distant boards degrade badly.
• Out-of-domain footage. On small, blurred, near-edge-on boards, piece scores saturate on people and background and the board box can come back an order of magnitude too large. The repository's guards (board gating, class-agnostic NMS, a 32-piece cap) bound the damage to something readable; none of them make it correct.
• Calibration is domain-specific. Fitted on ChessReD-like photographs; the further your images are from those, the less the confidences mean. Re-fit with chesssight train calibrate.
• Single seed. Every number here is from one training run. Differences under about 0.02 mAP are not distinguishable from noise.

Training

train5: 20 000 Blender/Cycles renders at 640×640. Positions 70% from real Lichess games / 30% uniform random; two chess sets (baked Staunton OBJ and procedural lathe profiles with silhouette taper); plastic, procedural wood/marble and photographed Poly Haven PBR materials with randomised hue, saturation and brightness, under a minimum piece-vs-square contrast floor; HDRI lighting and backdrop on every image; chess clocks in 45% of scenes plus up to three distractors; camera azimuth 0–360°, elevation 8–75°, 24–85 mm.

16 epochs, batch 12, AdamW, lr 1e-4 (backbone 1e-5), cosine schedule, classification loss weight 3.0, augmentation, EMA (0.9999). 147 min on one RTX 5070 Ti.

Intended use

Research and analysis on chess imagery. Not validated for officiating, rating, or any setting where a misread board carries a cost.

Licence

MIT. Training data generated from CC0 assets (Poly Haven HDRIs and textures) and public Lichess game dumps.