SegFormer (MiT-B5) fine-tuned on CoralscapesV2 (39 classes)

SegFormer model with a MiT-B5 backbone fine-tuned on CoralscapesV2 at resolution 1024x1024, on the 39-class (legacy Coralscapes) label set. Introduced in CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs.

Model Details

• Model type: SegFormer (semantic segmentation)
• Finetuned from: SegFormer (b5-sized) encoder nvidia/mit-b5
• Label set: 39 classes (model outputs 40 logits, ID 0 = void/ignored)
• Task: dense semantic segmentation of coral-reef imagery

Model Sources

• Training code: coralscapesScripts
• Paper: CoralscapesV2 (OpenReview)
• Project page: josauder.github.io/coralscapesv2
• Dataset: josauder/coralscapesV2

How to Get Started with the Model

The simplest way to segment an image is a single forward pass:

from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
from datasets import load_dataset

# Load an image from CoralscapesV2 (or use your own)
image = load_dataset("josauder/coralscapesV2", split="test")[0]["image"]

processor = SegformerImageProcessor.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024")
model = SegformerForSemanticSegmentation.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
labels = processor.post_process_semantic_segmentation(
    outputs, target_sizes=[(image.size[1], image.size[0])]
)[0].numpy()  # HxW array of class IDs

For images that are far from the 1024x1024 training resolution (e.g. the native 1024x2048 CoralscapesV2 frames), a strided sliding-window gives better results. This reproduces the strided 2 x 1024x1024 evaluation protocol reported below:

import torch
import torch.nn.functional as F
import numpy as np
from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
from datasets import load_dataset

device = "cuda" if torch.cuda.is_available() else "cpu"


def resize_image(image, target_size=1024):
    """Resize so that the shorter side equals `target_size`."""
    h_img, w_img = image.size
    if h_img < w_img:
        new_h, new_w = target_size, int(w_img * (target_size / h_img))
    else:
        new_h, new_w = int(h_img * (target_size / w_img)), target_size
    return image.resize((new_h, new_w))


def segment_image(image, processor, model, crop_size=(1024, 1024), num_classes=40):
    """Overlapping sliding-window inference with 1024x1024 windows."""
    h_crop, w_crop = crop_size
    img = torch.Tensor(np.array(resize_image(image, 1024)).transpose(2, 0, 1)).unsqueeze(0)
    batch_size, _, h_img, w_img = img.size()

    h_grids = int(np.round(3 / 2 * h_img / h_crop)) if h_img > h_crop else 1
    w_grids = int(np.round(3 / 2 * w_img / w_crop)) if w_img > w_crop else 1
    h_stride = int((h_img - h_crop + h_grids - 1) / (h_grids - 1)) if h_grids > 1 else h_crop
    w_stride = int((w_img - w_crop + w_grids - 1) / (w_grids - 1)) if w_grids > 1 else w_crop

    preds = img.new_zeros((batch_size, num_classes, h_img, w_img))
    count_mat = img.new_zeros((batch_size, 1, h_img, w_img))
    for h_idx in range(h_grids):
        for w_idx in range(w_grids):
            y1 = min(h_idx * h_stride + h_crop, h_img) - h_crop
            x1 = min(w_idx * w_stride + w_crop, w_img) - w_crop
            y2, x2 = y1 + h_crop, x1 + w_crop
            crop_img = img[:, :, y1:y2, x1:x2]
            inputs = processor(crop_img, return_tensors="pt")
            inputs["pixel_values"] = inputs["pixel_values"].to(device)
            with torch.no_grad():
                logits = model(**inputs).logits
            logits = F.interpolate(logits, size=crop_img.shape[-2:], mode="bilinear", align_corners=False)
            preds += F.pad(logits, (x1, preds.shape[3] - x2, y1, preds.shape[2] - y2)).cpu()
            count_mat[:, :, y1:y2, x1:x2] += 1
    preds = preds / count_mat
    preds = preds.argmax(dim=1)
    preds = F.interpolate(preds.unsqueeze(0).float(), size=image.size[::-1], mode="nearest")
    return preds.squeeze().cpu().numpy().astype(np.uint8)


processor = SegformerImageProcessor.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024")
model = SegformerForSemanticSegmentation.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024").to(device)
image = load_dataset("josauder/coralscapesV2", split="test")[0]["image"]
labels = segment_image(image, processor, model)

Training Details

Training Data

The model is trained on the CoralscapesV2 dataset, a general-purpose dense semantic segmentation dataset for coral reefs (2433 densely annotated 1024x2048 video frames from 55 reef sites in the Red Sea region).

Training Procedure

Following the best configuration of the original Coralscapes SegFormer baseline: AdamW optimizer with an initial learning rate of 6e-5, weight decay 1e-2, and a polynomial (power 1) learning-rate schedule for a fixed 100 epochs with batch size 4 on the combined train+val split. Augmentations: random resized crop to 1024x1024 (scale 0.02-1.0), color jitter (brightness/contrast/saturation in [0.8, 1.2], hue in [-0.05, 0.05]), random rotation up to +/-15 degrees, and random horizontal flips. Inputs are normalized with the ImageNet mean and standard deviation.

Evaluation

Results on the CoralscapesV2 and Coralscapes (V1) test splits, evaluated over the 39-class label set. single-pass predicts once at 768x1376; strided runs a 1024x1024 window at a 1024px stride (2 windows per frame), matching the protocol of the original Coralscapes paper. Metrics are pixel accuracy and mean IoU (%).

Classes and colors

The model outputs 40 logits (IDs 0-39). ID 0 is void / unlabeled and is ignored during training and evaluation; IDs 1-39 are the legacy Coralscapes (V1) semantic classes. fish is class 9.

The colors below are the official CoralscapesV2 visualization palette. The full id2label and label2color dictionaries are also distributed with the dataset as id2label.json and label2color.json.

{
  "0": "unlabeled",
  "1": "seagrass",
  "2": "trash",
  "3": "other coral dead",
  "4": "other coral bleached",
  "5": "sand",
  "6": "other coral alive",
  "7": "human",
  "8": "transect tools",
  "9": "fish",
  "10": "algae covered substrate",
  "11": "other animal",
  "12": "unknown hard substrate",
  "13": "background",
  "14": "dark",
  "15": "transect line",
  "16": "massive/meandering bleached",
  "17": "massive/meandering alive",
  "18": "rubble",
  "19": "branching bleached",
  "20": "branching dead",
  "21": "millepora",
  "22": "branching alive",
  "23": "massive/meandering dead",
  "24": "clam",
  "25": "acropora alive",
  "26": "sea cucumber",
  "27": "turbinaria",
  "28": "table acropora alive",
  "29": "sponge",
  "30": "anemone",
  "31": "pocillopora alive",
  "32": "table acropora dead",
  "33": "meandering bleached",
  "34": "stylophora alive",
  "35": "sea urchin",
  "36": "meandering alive",
  "37": "meandering dead",
  "38": "crown of thorn",
  "39": "dead clam"
}

{
  "seagrass": [
    125,
    222,
    125
  ],
  "trash": [
    255,
    0,
    134
  ],
  "other coral dead": [
    114,
    60,
    61
  ],
  "other coral bleached": [
    250,
    224,
    225
  ],
  "sand": [
    194,
    178,
    128
  ],
  "other coral alive": [
    224,
    118,
    119
  ],
  "human": [
    255,
    0,
    0
  ],
  "transect tools": [
    8,
    205,
    12
  ],
  "fish": [
    255,
    255,
    0
  ],
  "algae covered substrate": [
    125,
    163,
    125
  ],
  "other animal": [
    0,
    255,
    255
  ],
  "unknown hard substrate": [
    125,
    125,
    125
  ],
  "background": [
    29,
    162,
    216
  ],
  "dark": [
    31,
    31,
    31
  ],
  "transect line": [
    0,
    255,
    0
  ],
  "massive/meandering bleached": [
    255,
    248,
    228
  ],
  "massive/meandering alive": [
    236,
    150,
    21
  ],
  "rubble": [
    161,
    153,
    128
  ],
  "branching bleached": [
    252,
    231,
    240
  ],
  "branching dead": [
    123,
    50,
    86
  ],
  "millepora": [
    244,
    150,
    115
  ],
  "branching alive": [
    226,
    91,
    157
  ],
  "massive/meandering dead": [
    134,
    86,
    18
  ],
  "clam": [
    189,
    255,
    234
  ],
  "acropora alive": [
    236,
    128,
    255
  ],
  "sea cucumber": [
    0,
    231,
    255
  ],
  "turbinaria": [
    228,
    255,
    119
  ],
  "table acropora alive": [
    189,
    119,
    255
  ],
  "sponge": [
    240,
    80,
    80
  ],
  "anemone": [
    0,
    255,
    189
  ],
  "pocillopora alive": [
    255,
    146,
    150
  ],
  "table acropora dead": [
    85,
    53,
    116
  ],
  "meandering bleached": [
    251,
    243,
    216
  ],
  "stylophora alive": [
    255,
    111,
    194
  ],
  "sea urchin": [
    0,
    142,
    255
  ],
  "meandering alive": [
    230,
    193,
    0
  ],
  "meandering dead": [
    119,
    100,
    14
  ],
  "crown of thorn": [
    179,
    245,
    234
  ],
  "dead clam": [
    89,
    155,
    134
  ]
}

Citation

If you find this model useful, please cite:

@inproceedings{sauder2026coralscapesv2,
  title={CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Ruckli, Thomas and Strodomskyt{\.e}, Gabriel{\.e} and Abdallah, Ibrahim Souleiman and Abdi, Rahma Hassan and Awaleh, Djama Goumaneh and Farah, Mohamed Houssein and Nour, Moustapha and Saad, Osama Sharhubil and Altaib, Mustafa Mohammed Khalafallah and Kteifan, Maysoon and Alsoqi, Farah and Zgool, Eyad and Al-Omari, Jafar and Gebreluel, Temesgen Gebremeskel and Abdulkerim, Zekaria Zekeria and Ghirmay, Meron and Beraki, Teklehaimanot and Tuia, Devis and Banc-Prandi, Guilhem},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV) Workshop on Marine Vision},
  year={2026}
}

The dataset builds on the original Coralscapes dataset:

@inproceedings{sauder2025coralscapes,
  title={The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Domazetoski, Viktor and Banc-Prandi, Guilhem and Perna, Gabriela and Meibom, Anders and Tuia, Devis},
  booktitle={Proceedings of the International Conference on Computer Vision (ICCV) Joint Workshop on Marine Vision},
  year={2025}
}

License

Released under the Apache License 2.0.