pi-r2 — DETR fine-tuned on Beyond Words (LOC)
Object detection model fine-tuned from facebook/detr-resnet-50
(Apache-2.0) on the biglam/loc_beyond_words
dataset (CC0): crowd-sourced bounding-box annotations of World War I-era newspaper pages
from the Library of Congress Chronicling America collection.
Model detail
- Architecture: DETR (DEtection TRansformer) with a ResNet-50 backbone, 6 encoder/6 decoder transformer layers, 100 object queries.
- Base model:
facebook/detr-resnet-50— license Apache-2.0 (shareable). - Dataset:
biglam/loc_beyond_words— license CC0-1.0 (public domain). - Classes (7): Photograph, Illustration, Map, Comics/Cartoon, Editorial Cartoon, Headline, Advertisement
- Image size: resized so the longest edge ≤ 1200 px (aspect ratio preserved),
padded per batch via a
pixel_mask. Training used random horizontal flips and random scale (70–100% of the max size). - Optimizer: AdamW (LR 5e-05, weight decay 0.0001), cosine schedule with 5% warmup, fp16, gradient clipping 0.1.
- Class imbalance: images containing rare classes (Map, Editorial Cartoon, Illustration, Comics)
are oversampled with a
WeightedRandomSamplerwhen building training batches. - Training budget: 0.19 minutes, 24 steps, batch size 4 on an NVIDIA A10G.
Validation results (COCO protocol, pycocotools)
Evaluated on the biglam/loc_beyond_words validation split (32 images)
with COCO-style IoU-matched metrics (area = all, max detections = 300, NMS IoU threshold 0.75).
| Metric | Value |
|---|---|
| mAP @[0.5:0.95] | 1.843759794973911e-06 |
| mAP @0.50 | 6.675969148985493e-05 |
| mAP @0.75 | 0.0 |
| AR @100 | 0.0021307798481711524 |
Per-class AP @0.50:
| Class | AP@0.50 |
|---|---|
| Photograph | - |
Raw per-image predictions and this training script are included in this repo (
eval_predictions.json,train_detr.py).
Usage
from transformers import AutoImageProcessor, DetrForObjectDetection
from PIL import Image
import torch
repo = "harness-race/pi-r2-probe"
processor = AutoImageProcessor.from_pretrained(repo)
model = DetrForObjectDetection.from_pretrained(repo)
img = Image.open("newspaper_page.jpg").convert("RGB")
inputs = processor(images=img, return_tensors="pt")
with torch.no_grad():
out = model(**inputs)
score_threshold = 0.5
for logits, box in zip(out.logits[0], out.pred_boxes[0]):
prob = logits.softmax(-1)
cls_idx, score = prob[:, :-1].max(-1)
if score.item() > score_threshold:
cx, cy, bw, bh = box.tolist()
W, H = img.size
x1, y1 = (cx - bw/2)*W, (cy - bh/2)*H
x2, y2 = (cx + bw/2)*W, (cy + bh/2)*H
print(model.config.id2label[cls_idx.item()], round(score.item(), 3), [round(x1), round(y1), round(x2), round(y2)])
Intended use & limitations
- Trained on historically scanned newspaper pages (circa 1910–1920); other domains or modern document layouts will degrade accuracy.
- The dataset is strongly class-imbalanced (Headline/Advertisement dominate; Map and Editorial Cartoon are rare), so per-class accuracy varies widely (see table above).
- DETR emits up to 100 box proposals per image; in very dense pages some objects may be missed.
Licenses
- Base model
facebook/detr-resnet-50: Apache-2.0. - Dataset
biglam/loc_beyond_words: CC0-1.0 (Public Domain Dedication). - This fine-tuned model: Apache-2.0.