Run Grounding DINO with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/grounding_dino_base

Paper: Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (arXiv:2303.05499) · HF Papers

Grounding DINO performs open-set, text-grounded object detection: it finds the objects a free-form text prompt names, not a fixed label set. A Swin image backbone and a BERT text encoder feed a deformable cross-modality encoder that fuses vision and language, a contrastive query-selection stage picks object proposals, and a decoder with iterative box refinement emits one box per query scored against the prompt tokens. No anchors, no NMS, and categories that were never in a detection training set (here "Swin-Base" backbone).

For more details on the model, please go to IDEA-Research's original model card.

Pure-Keras 3 conversion of IDEA-Research/grounding-dino-base for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is an open-set object detection checkpoint (GroundingDinoForObjectDetection, Swin-Base backbone): each query predicts a box and a score over the prompt tokens.

✨ Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import torch
from PIL import Image
from zeromodels.models.grounding_dino import (
    GroundingDinoForObjectDetection,
    GroundingDinoProcessor,
)

model = GroundingDinoForObjectDetection.from_weights("zeromodels/grounding_dino_base")
processor = GroundingDinoProcessor.from_weights("zeromodels/grounding_dino_base")

image = Image.open("your_image.jpg").convert("RGB")
# Prompts are free text; pass a list of candidates (or one "a. b. c." string). Skip
# articles: in "a paddle" the "a" can outscore the noun.
inputs = processor(images=image, text=["person", "paddle", "board"])

with torch.no_grad():  # torch backend: avoids a large autograd graph (can OOM otherwise)
    output = model(inputs)
# output["logits"]:     (1, 900, 256)
# output["pred_boxes"]: (1, 900, 4)

results = processor.post_process_object_detection(
    output,
    threshold=0.3,
    target_sizes=[(image.height, image.width)],
    input_ids=inputs["input_ids"],
)[0]
for score, name, box in sorted(
    zip(results["scores"], results["text_labels"], results["boxes"]),
    key=lambda d: -float(d[0]),
):
    print(f"{name}: {float(score):.3f} {[round(float(v)) for v in box]}")

Load either Grounding DINO variant the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• On the torch backend, wrap inference in with torch.no_grad(): — the forward keeps a large autograd graph otherwise and can OOM. The JAX / TensorFlow backends need no such wrap.
• Write prompts as lower-case phrases separated as a list or by .; drop articles ("a", "the") so the noun scores highest. post_process_object_detection needs input_ids= to map scores back to prompt words (text_labels).
• threshold=0.3 is a reasonable start; raise it for cleaner scenes.
• See Grounding DINO docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. GroundingDinoForObjectDetection.from_weights("hf:IDEA-Research/grounding-dino-base").

Special Thanks

A huge thank you to the IDEA-Research authors for creating and releasing Grounding DINO.

License: Apache 2.0.