DETR (ResNet-50) fine-tuned on Beyond Words (loc_beyond_words)
Object detection model detecting 7 types of visual content on World War I-era historical newspaper pages (Library of Congress, Chronicling America): photographs, illustrations, maps, comics/cartoons, editorial cartoons, headlines, and advertisements.
- Base model:
facebook/detr-resnet-50(Apache-2.0) - Dataset:
biglam/loc_beyond_words(CC0-1.0), 2,846 train / 712 validation images, COCO format - Task: object detection (bounding boxes + class labels)
How to use
from transformers import AutoImageProcessor, DetrForObjectDetection
from PIL import Image
import requests
url = "https://tile.loc.gov/storage-services/service/amed/amedscd/2018603128/001.jpg" # example page
image = Image.open(requests.get(url, stream=True).raw)
processor = AutoImageProcessor.from_pretrained("harness-race/pi-r3")
model = DetrForObjectDetection.from_pretrained("harness-race/pi-r3")
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
target_sizes = [(image.height, image.width)]
results = processor.post_process_object_detection(outputs, threshold=0.5,
target_sizes=target_sizes)[0]
for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
print(score, model.config.id2label[label.item()], [round(x, 2) for x in box.tolist()])
Or with the pipeline:
from transformers import pipeline
detector = pipeline("object-detection", model="harness-race/pi-r3")
results = detector(image, threshold=0.5)
Training details
Fine-tuned from the COCO-pretrained checkpoint with the HF Transforms API (aspect-preserving resize to shortest edge 480px, max 768px, padding with pixel masks, random horizontal flip augmentation).
| Hyperparameter | Value |
|---|---|
| Epochs (best) | 13 of 14 requested |
| Batch size | 4 |
| Optimizer | AdamW |
| Learning rate (main / backbone) | 5e-05 / 5e-06 |
| Weight decay | 0.0001 |
| Warmup / schedule | 2% linear warmup + cosine decay |
| Gradient clipping | 10.0 |
| Early stopping patience | 3 (on validation loss) |
| Max input size | 768px |
| Total training time | 57 min |
Class distribution in the training set is skewed (Headline ~58%, Advertisement ~28%, followed by Photograph / Comics / Illustration / Editorial Cartoon / Map).
Validation results (hold-out split, COCO protocol)
| Metric | Value |
|---|---|
| mAP (IoU 0.5:0.95) | 0.1130 |
| mAP@0.5 | 0.2466 |
| mAP@0.75 | 0.0954 |
| mAR (max 100 dets) | 0.2892 |
Per-class mAP (IoU 0.5:0.95, area=all, maxDets=100):
| Class | mAP |
|---|---|
| Photograph | 0.2316 |
| Illustration | 0.0531 |
| Map | 0.0189 |
| Comics/Cartoon | 0.1403 |
| Editorial Cartoon | 0.1589 |
| Headline | 0.0650 |
| Advertisement | 0.1236 |
Training/validation loss curves and full eval details are in training_log.json
in this repository.
Limitations
- Trained on WWI-era (mostly 1910s) American newspaper scans; may transfer less well to other layouts, eras, or languages.
- Classes are heavily imbalanced (headlines and advertisements dominate).
- Input resolution is limited to 768px on the longest edge; re-running inference at higher resolution may improve small-object recall.
- Model weights are the DETR Apache-2.0-licensed architecture; dataset is CC0.