Surgical SAM 3.1 V2

Surgical SAM 3.1 V2 is an open vocabulary detector and instance segmenter for surgical scenes. A text prompt specifies an instrument, instrument part, organ, or tissue category. The dataset schema defines 33 prompt categories, of which 29 have positive training annotations.

V2 is trained from Base SAM 3.1 on 34,316 training images. The expanded data add (i) liver and gallbladder masks from CRCD and (ii) instrument body, wrist, and gripper masks from uTenn. Each image records its annotation scope, so training and evaluation issue negative labels only for categories that were annotated in that image.

The default checkpoint combines the fine tuned image detector with the native SAM 3.1 multiplex tracker. The detector initializes text prompted objects, and the tracker propagates their identities through video.

Results

We selected the checkpoint using mask AP on the 4,248 image validation split. We then evaluated it once on the expanded test split, which contains 4,320 images and 28 categories with positive instances.

Comparison with Surgical SAM 3.1

The matched comparison uses the original grouped test split, which contains 4,134 images and 24 categories with positive instances. Surgical SAM 3.1 V2 and the previous Surgical SAM 3.1 model use identical images, prompts, annotation scopes, postprocessing, and COCO evaluation settings.

COCO AP averages precision across ten intersection over union thresholds from 0.50 through 0.95. The expanded result and the matched comparison use different test populations, so they are reported separately.

Category groups

Model selection

The full model was trained for eight epochs from Base SAM 3.1. Epoch 7 had the highest validation mask AP among the full model checkpoints. We used that checkpoint to train the segmentation head for one epoch with the Dice loss configuration. A final epoch, which used a lower learning rate, produced a similar result. We selected the Dice refinement because it had the highest validation mask AP.

Evaluation protocol

Each image contains exhaustive_category_ids, which lists the categories that are annotated in that image. For each image, the evaluation script runs inference with the categories in exhaustive_category_ids and scores the resulting predictions with the COCO evaluator. Categories outside that list do not contribute positive or negative examples for the image.

The merged split assigns each procedure, video, or patient group to one split. The original grouped test subset is preserved inside the expanded test split, which permits the matched comparison above.

Quick start

Download the image model builder patch:

hf download mehmetkeremturkcan/Surgical-SAM-3.1-V2 \
  patches/sam31_image_training.patch \
  --revision main \
  --local-dir surgical_sam31_v2

Install the SAM 3 source used for this run:

git clone https://github.com/facebookresearch/sam3.git
cd sam3
git checkout 46957e47805eaa273f4aa7bbbd25a88bca9108ce
git apply ../surgical_sam31_v2/patches/sam31_image_training.patch
pip install -e .

Download the checkpoint and configuration:

hf download mehmetkeremturkcan/Surgical-SAM-3.1-V2 \
  config.json \
  joint_grouped_sam31_v2_best_multiplex.pt \
  --revision main \
  --local-dir checkpoints

Run image detection and segmentation:

import torch
from PIL import Image

from sam3.model.sam3_image_processor import Sam3Processor
from sam3.model_builder import build_sam31_image_model

model = build_sam31_image_model(
    checkpoint_path="checkpoints/joint_grouped_sam31_v2_best_multiplex.pt",
    load_from_HF=False,
    device="cuda",
    eval_mode=True,
    enable_segmentation=True,
)
processor = Sam3Processor(model, device="cuda", confidence_threshold=0.5)
image = Image.open("frame.png").convert("RGB")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = processor.set_image(image)
    output = processor.set_text_prompt(
        state=state,
        prompt="surgical instrument",
    )

boxes = output["boxes"]
masks = output["masks"]
scores = output["scores"]

The complete example is available in scripts/inference_sam31.py.

Checkpoints

Both checkpoints load with weights_only=True and contain zero unsafe serialization globals. The multiplex checkpoint contains 1,166 detector tensors and 457 tracker tensors. Its tracker tensors retain the values from facebook/sam3.1.

Training

Dataset

The V2 dataset contains 42,884 unique images and 188,229 annotations. Its schema defines 33 prompt categories, of which 29 have positive training annotations. Train, validation, and test contain 34,316, 4,248, and 4,320 images. The V2 package will be released separately in the associated dataset repository.

Release video

The video uses synthetic benchtop footage and mask only predictions for anatomy and instrument parts.

Intended use

This repository supports research in surgical computer vision and robotics. The model has not been validated for clinical decision making or patient care.

Authors

Mehmet Kerem Turkcan, Soham Samal, and Zoran Kostic. This project was developed by the AIDL Lab at Columbia University and IPAL Lab.

Citation

@misc{turkcan2026surgicalsam31v2,
  title  = {Surgical SAM 3.1 V2},
  author = {Turkcan, Mehmet Kerem and Samal, Soham and Kostic, Zoran},
  year   = {2026},
  url    = {https://huggingface.co/mehmetkeremturkcan/Surgical-SAM-3.1-V2}
}

Licenses

The checkpoints follow the SAM License in LICENSE. Dataset records retain their source licenses. The dataset repository lists each source license and attribution.