SegFormer (MiT-B5) fine-tuned on CoralscapesV2 (95 classes)

SegFormer model with a MiT-B5 backbone fine-tuned on CoralscapesV2 at resolution 1024x1024, on the fine-grained 95-class label set. Introduced in CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs.

Model Details

• Model type: SegFormer (semantic segmentation)
• Finetuned from: SegFormer (b5-sized) encoder nvidia/mit-b5
• Label set: 95 classes (model outputs 96 logits, ID 0 = void/ignored)
• Task: dense semantic segmentation of coral-reef imagery

Model Sources

• Training code: coralscapesScripts
• Paper: CoralscapesV2 (OpenReview)
• Project page: josauder.github.io/coralscapesv2
• Dataset: josauder/coralscapesV2

How to Get Started with the Model

The simplest way to segment an image is a single forward pass:

from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
from datasets import load_dataset

# Load an image from CoralscapesV2 (or use your own)
image = load_dataset("josauder/coralscapesV2", split="test")[0]["image"]

processor = SegformerImageProcessor.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024-95_class")
model = SegformerForSemanticSegmentation.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024-95_class")

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
labels = processor.post_process_semantic_segmentation(
    outputs, target_sizes=[(image.size[1], image.size[0])]
)[0].numpy()  # HxW array of class IDs

For images that are far from the 1024x1024 training resolution (e.g. the native 1024x2048 CoralscapesV2 frames), a strided sliding-window gives better results. This reproduces the strided 2 x 1024x1024 evaluation protocol reported below:

import torch
import torch.nn.functional as F
import numpy as np
from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation
from datasets import load_dataset

device = "cuda" if torch.cuda.is_available() else "cpu"


def resize_image(image, target_size=1024):
    """Resize so that the shorter side equals `target_size`."""
    h_img, w_img = image.size
    if h_img < w_img:
        new_h, new_w = target_size, int(w_img * (target_size / h_img))
    else:
        new_h, new_w = int(h_img * (target_size / w_img)), target_size
    return image.resize((new_h, new_w))


def segment_image(image, processor, model, crop_size=(1024, 1024), num_classes=96):
    """Overlapping sliding-window inference with 1024x1024 windows."""
    h_crop, w_crop = crop_size
    img = torch.Tensor(np.array(resize_image(image, 1024)).transpose(2, 0, 1)).unsqueeze(0)
    batch_size, _, h_img, w_img = img.size()

    h_grids = int(np.round(3 / 2 * h_img / h_crop)) if h_img > h_crop else 1
    w_grids = int(np.round(3 / 2 * w_img / w_crop)) if w_img > w_crop else 1
    h_stride = int((h_img - h_crop + h_grids - 1) / (h_grids - 1)) if h_grids > 1 else h_crop
    w_stride = int((w_img - w_crop + w_grids - 1) / (w_grids - 1)) if w_grids > 1 else w_crop

    preds = img.new_zeros((batch_size, num_classes, h_img, w_img))
    count_mat = img.new_zeros((batch_size, 1, h_img, w_img))
    for h_idx in range(h_grids):
        for w_idx in range(w_grids):
            y1 = min(h_idx * h_stride + h_crop, h_img) - h_crop
            x1 = min(w_idx * w_stride + w_crop, w_img) - w_crop
            y2, x2 = y1 + h_crop, x1 + w_crop
            crop_img = img[:, :, y1:y2, x1:x2]
            inputs = processor(crop_img, return_tensors="pt")
            inputs["pixel_values"] = inputs["pixel_values"].to(device)
            with torch.no_grad():
                logits = model(**inputs).logits
            logits = F.interpolate(logits, size=crop_img.shape[-2:], mode="bilinear", align_corners=False)
            preds += F.pad(logits, (x1, preds.shape[3] - x2, y1, preds.shape[2] - y2)).cpu()
            count_mat[:, :, y1:y2, x1:x2] += 1
    preds = preds / count_mat
    preds = preds.argmax(dim=1)
    preds = F.interpolate(preds.unsqueeze(0).float(), size=image.size[::-1], mode="nearest")
    return preds.squeeze().cpu().numpy().astype(np.uint8)


processor = SegformerImageProcessor.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024-95_class")
model = SegformerForSemanticSegmentation.from_pretrained("EPFL-ECEO/segformer-b5-finetuned-coralscapesv2-1024-1024-95_class").to(device)
image = load_dataset("josauder/coralscapesV2", split="test")[0]["image"]
labels = segment_image(image, processor, model)

Training Details

Training Data

The model is trained on the CoralscapesV2 dataset, a general-purpose dense semantic segmentation dataset for coral reefs (2433 densely annotated 1024x2048 video frames from 55 reef sites in the Red Sea region).

Training Procedure

Following the best configuration of the original Coralscapes SegFormer baseline: AdamW optimizer with an initial learning rate of 6e-5, weight decay 1e-2, and a polynomial (power 1) learning-rate schedule for a fixed 100 epochs with batch size 4 on the combined train+val split. Augmentations: random resized crop to 1024x1024 (scale 0.02-1.0), color jitter (brightness/contrast/saturation in [0.8, 1.2], hue in [-0.05, 0.05]), random rotation up to +/-15 degrees, and random horizontal flips. Inputs are normalized with the ImageNet mean and standard deviation.

Evaluation

Results on the CoralscapesV2 and Coralscapes (V1) test splits. This model is trained natively on the 95-class set; we report the native 95-class metrics as well as metrics obtained by mapping the 95-class predictions to the 39-class legacy set for comparison. single-pass predicts once at 768x1376; strided runs a 1024x1024 window at a 1024px stride (2 windows per frame). Metrics are pixel accuracy and mean IoU (%).

Classes and colors

The model outputs 96 logits (IDs 0-95). ID 0 is void / unlabeled and is ignored during training and evaluation; IDs 1-95 are the CoralscapesV2 semantic classes. fish is class 25.

The colors below are the official CoralscapesV2 visualization palette. The full id2label and label2color dictionaries are also distributed with the dataset as id2label.json and label2color.json.

{
  "0": "unlabeled",
  "1": "acanthaster planci",
  "2": "acropora alive",
  "3": "acropora bleached",
  "4": "acropora dead",
  "5": "algae covered brain coral",
  "6": "algae covered branching coral",
  "7": "algae covered lobophylliidae",
  "8": "algae covered massive coral",
  "9": "algae covered pocillopora",
  "10": "algae covered porites",
  "11": "algae covered substrate",
  "12": "algae covered table acropora",
  "13": "anemone",
  "14": "background",
  "15": "brain coral alive",
  "16": "brain coral bleached",
  "17": "brain coral dead",
  "18": "branching coral alive",
  "19": "branching coral bleached",
  "20": "branching coral dead",
  "21": "cirrhipathes",
  "22": "dark",
  "23": "dead giant clam",
  "24": "feather worm",
  "25": "fish",
  "26": "fungiidae alive",
  "27": "fungiidae bleached",
  "28": "fungiidae dead",
  "29": "galaxea alive",
  "30": "galaxea bleached",
  "31": "galaxea dead",
  "32": "giant clam",
  "33": "goniopora alive",
  "34": "goniopora dead",
  "35": "hard coral alive",
  "36": "hard coral bleached",
  "37": "hard coral dead",
  "38": "hard substrate",
  "39": "human",
  "40": "human-made structure",
  "41": "leather coral alive",
  "42": "lobophylliidae alive",
  "43": "lobophylliidae bleached",
  "44": "lobophylliidae dead",
  "45": "massive alive",
  "46": "massive bleached",
  "47": "massive dead",
  "48": "meandering coral alive",
  "49": "meandering coral bleached",
  "50": "meandering coral dead",
  "51": "millepora",
  "52": "millepora bleached",
  "53": "millepora dead",
  "54": "nephtheidae alive",
  "55": "other animal",
  "56": "other coral alive",
  "57": "other coral bleached",
  "58": "other coral dead",
  "59": "pavona alive",
  "60": "pavona bleached",
  "61": "pavona dead",
  "62": "pocillopora alive",
  "63": "pocillopora bleached",
  "64": "pocillopora dead",
  "65": "porites alive",
  "66": "porites bleached",
  "67": "porites dead",
  "68": "rubble",
  "69": "sand",
  "70": "sea cucumber",
  "71": "sea urchin",
  "72": "seagrass",
  "73": "seriatopora alive",
  "74": "seriatopora bleached",
  "75": "seriatopora dead",
  "76": "soft coral (malacalcyonacea) alive",
  "77": "soft coral (malacalcyonacea) bleached",
  "78": "sponge",
  "79": "stylophora alive",
  "80": "stylophora bleached",
  "81": "stylophora dead",
  "82": "table acropora alive",
  "83": "table acropora bleached",
  "84": "table acropora dead",
  "85": "thin plate/encrusting coral alive",
  "86": "thin plate/encrusting coral bleached",
  "87": "thin plate/encrusting coral dead",
  "88": "transect line",
  "89": "transect tools",
  "90": "trash",
  "91": "turbinaria",
  "92": "turbinaria dead",
  "93": "water column",
  "94": "water surface",
  "95": "xeniidae alive"
}

{
  "acanthaster planci": [
    179,
    245,
    234
  ],
  "acropora alive": [
    236,
    128,
    255
  ],
  "acropora bleached": [
    255,
    236,
    254
  ],
  "acropora dead": [
    118,
    63,
    128
  ],
  "algae covered brain coral": [
    208,
    207,
    58
  ],
  "algae covered branching coral": [
    6,
    71,
    245
  ],
  "algae covered lobophylliidae": [
    50,
    208,
    130
  ],
  "algae covered massive coral": [
    67,
    182,
    208
  ],
  "algae covered pocillopora": [
    225,
    109,
    77
  ],
  "algae covered porites": [
    173,
    81,
    130
  ],
  "algae covered substrate": [
    125,
    163,
    125
  ],
  "algae covered table acropora": [
    48,
    162,
    221
  ],
  "anemone": [
    0,
    255,
    189
  ],
  "background": [
    29,
    162,
    216
  ],
  "brain coral alive": [
    114,
    78,
    120
  ],
  "brain coral bleached": [
    219,
    67,
    238
  ],
  "brain coral dead": [
    78,
    172,
    99
  ],
  "branching coral alive": [
    226,
    91,
    157
  ],
  "branching coral bleached": [
    252,
    231,
    240
  ],
  "branching coral dead": [
    123,
    50,
    86
  ],
  "cirrhipathes": [
    6,
    189,
    22
  ],
  "dark": [
    31,
    31,
    31
  ],
  "dead giant clam": [
    89,
    155,
    134
  ],
  "feather worm": [
    159,
    186,
    38
  ],
  "fish": [
    255,
    255,
    0
  ],
  "fungiidae alive": [
    100,
    133,
    248
  ],
  "fungiidae bleached": [
    200,
    138,
    27
  ],
  "fungiidae dead": [
    3,
    92,
    44
  ],
  "galaxea alive": [
    96,
    188,
    108
  ],
  "galaxea bleached": [
    67,
    243,
    60
  ],
  "galaxea dead": [
    176,
    225,
    11
  ],
  "giant clam": [
    189,
    255,
    234
  ],
  "goniopora alive": [
    181,
    5,
    120
  ],
  "goniopora dead": [
    179,
    191,
    30
  ],
  "hard coral alive": [
    109,
    232,
    188
  ],
  "hard coral bleached": [
    70,
    247,
    149
  ],
  "hard coral dead": [
    114,
    15,
    203
  ],
  "hard substrate": [
    125,
    125,
    125
  ],
  "human": [
    255,
    0,
    0
  ],
  "human-made structure": [
    151,
    171,
    80
  ],
  "leather coral alive": [
    86,
    184,
    94
  ],
  "lobophylliidae alive": [
    201,
    73,
    234
  ],
  "lobophylliidae bleached": [
    253,
    187,
    79
  ],
  "lobophylliidae dead": [
    57,
    106,
    220
  ],
  "massive alive": [
    236,
    150,
    21
  ],
  "massive bleached": [
    255,
    248,
    228
  ],
  "massive dead": [
    134,
    86,
    18
  ],
  "meandering coral alive": [
    230,
    193,
    0
  ],
  "meandering coral bleached": [
    251,
    243,
    216
  ],
  "meandering coral dead": [
    119,
    100,
    14
  ],
  "millepora": [
    244,
    150,
    115
  ],
  "millepora bleached": [
    164,
    226,
    52
  ],
  "millepora dead": [
    119,
    128,
    174
  ],
  "nephtheidae alive": [
    132,
    109,
    91
  ],
  "other animal": [
    0,
    255,
    255
  ],
  "other coral alive": [
    224,
    118,
    119
  ],
  "other coral bleached": [
    250,
    224,
    225
  ],
  "other coral dead": [
    114,
    60,
    61
  ],
  "pavona alive": [
    13,
    191,
    102
  ],
  "pavona bleached": [
    118,
    249,
    130
  ],
  "pavona dead": [
    141,
    130,
    175
  ],
  "pocillopora alive": [
    255,
    146,
    150
  ],
  "pocillopora bleached": [
    250,
    219,
    220
  ],
  "pocillopora dead": [
    130,
    66,
    66
  ],
  "porites alive": [
    255,
    209,
    71
  ],
  "porites bleached": [
    213,
    10,
    25
  ],
  "porites dead": [
    40,
    24,
    243
  ],
  "rubble": [
    161,
    153,
    128
  ],
  "sand": [
    194,
    178,
    128
  ],
  "sea cucumber": [
    0,
    231,
    255
  ],
  "sea urchin": [
    0,
    142,
    255
  ],
  "seagrass": [
    125,
    222,
    125
  ],
  "seriatopora alive": [
    191,
    19,
    6
  ],
  "seriatopora bleached": [
    247,
    89,
    191
  ],
  "seriatopora dead": [
    75,
    131,
    128
  ],
  "soft coral (malacalcyonacea) alive": [
    147,
    144,
    54
  ],
  "soft coral (malacalcyonacea) bleached": [
    7,
    227,
    247
  ],
  "sponge": [
    240,
    80,
    80
  ],
  "stylophora alive": [
    255,
    111,
    194
  ],
  "stylophora bleached": [
    255,
    220,
    241
  ],
  "stylophora dead": [
    123,
    54,
    94
  ],
  "table acropora alive": [
    189,
    119,
    255
  ],
  "table acropora bleached": [
    244,
    233,
    253
  ],
  "table acropora dead": [
    85,
    53,
    116
  ],
  "thin plate/encrusting coral alive": [
    91,
    28,
    190
  ],
  "thin plate/encrusting coral bleached": [
    42,
    209,
    57
  ],
  "thin plate/encrusting coral dead": [
    8,
    19,
    242
  ],
  "transect line": [
    0,
    255,
    0
  ],
  "transect tools": [
    8,
    205,
    12
  ],
  "trash": [
    255,
    0,
    134
  ],
  "turbinaria": [
    228,
    255,
    119
  ],
  "turbinaria dead": [
    169,
    30,
    70
  ],
  "water column": [
    31,
    0,
    166
  ],
  "water surface": [
    0,
    255,
    248
  ],
  "xeniidae alive": [
    139,
    142,
    168
  ]
}

Citation

If you find this model useful, please cite:

@inproceedings{sauder2026coralscapesv2,
  title={CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Ruckli, Thomas and Strodomskyt{\.e}, Gabriel{\.e} and Abdallah, Ibrahim Souleiman and Abdi, Rahma Hassan and Awaleh, Djama Goumaneh and Farah, Mohamed Houssein and Nour, Moustapha and Saad, Osama Sharhubil and Altaib, Mustafa Mohammed Khalafallah and Kteifan, Maysoon and Alsoqi, Farah and Zgool, Eyad and Al-Omari, Jafar and Gebreluel, Temesgen Gebremeskel and Abdulkerim, Zekaria Zekeria and Ghirmay, Meron and Beraki, Teklehaimanot and Tuia, Devis and Banc-Prandi, Guilhem},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV) Workshop on Marine Vision},
  year={2026}
}

The dataset builds on the original Coralscapes dataset:

@inproceedings{sauder2025coralscapes,
  title={The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Domazetoski, Viktor and Banc-Prandi, Guilhem and Perna, Gabriela and Meibom, Anders and Tuia, Devis},
  booktitle={Proceedings of the International Conference on Computer Vision (ICCV) Joint Workshop on Marine Vision},
  year={2025}
}

License

Released under the Apache License 2.0.