Detecture — ICLR 2027 checkpoint
The trained model behind every number reported in the ICLR 2027 submission on
sub-semantic image segmentation. This is the checkpoint referenced as
epoch_23.pt by fairness_baseline_suite/src/paths.py.
Code: github.com/aviadcohz/Qwen2SAM_Detecture_Benchmark Data: aviadcohz/Detecture_ICLR_Benchmarking
What this is
Detecture couples a general-purpose vision-language model to a promptable segmentation backbone. Given only an image and a single open-range instruction, it emits its own free-form texture descriptions, infers how many regions the scene contains, and resolves them into a non-overlapping partition.
- Qwen3-VL-8B reasons over appearance and emits one description per region,
each terminated by a
[SEG]grounding token. - A learned Bridge maps each 4096-d
[SEG]hidden state into SAM 3's native 1024-d text width. - SAM 3 is frozen throughout — image encoder, text resizer, transformer and mask decoder.
- A Winner-Takes-All assignment over K+1 channels, including a learned dustbin, turns the per-texture masks into a partition rather than a set of independently thresholded masks.
Trainable parameters
Only 8.23 M parameters are trained. The rest of both backbones is frozen.
| Component | Parameters |
|---|---|
| Bridge projector | 4.20 M |
| Qwen3-VL LoRA (attention q, v) | 3.83 M |
| Mask head | 0.20 M |
| Dustbin embedding | ~0 |
| Total | 8.23 M |
What is in the file
detecture_epoch23.pt (2.5 GB) holds:
model_trainable 157 tensors the 8.23 M trained parameters, plus the resized
Qwen embedding and LM head (1242 M, frozen — they
appear here only because adding the [SEG] token
resized the vocabulary)
dustbin_embed the learned dustbin vector
epoch, phase, val_iou training metadata
Optimizer state has been stripped: the original training checkpoint is 7.6 GB, and
training/utils.py::load_checkpoint only touches the optimizer key when an
optimizer is passed, so inference and evaluation are unaffected. Resuming training
from this file will reinitialise the AdamW moments.
Use
huggingface-cli download aviadcohz/Detecture-ICLR-2027 detecture_epoch23.pt \
--local-dir ~/Detecture_ICLR/Qwen2SAM_Detecture/checkpoints/
mv ~/Detecture_ICLR/Qwen2SAM_Detecture/checkpoints/detecture_epoch23.pt \
~/Detecture_ICLR/Qwen2SAM_Detecture/checkpoints/epoch_23.pt
That path is what paths.py resolves. Then:
cd Qwen2SAM_Detecture_Benchmark/fairness_baseline_suite
PYTHONPATH=src python src/run_fairness.py --model detecture --dataset RWTD
Requires the Qwen3-VL-8B and SAM 3 base weights; the loader copies these trained tensors into an assembled model rather than standing alone.
Reported results
Under a protocol in which no method is told how many regions an image contains:
| Route | Region Covering | Boundary-F | mIoU | ARI |
|---|---|---|---|---|
| RWTD | 0.7946 | 0.4252 | 0.7843 | 0.5993 |
| RWTD-COCO | 0.8536 | 0.4226 | 0.8460 | 0.7351 |
| TextureADE | 0.7479 | 0.5038 | 0.7222 | 0.6863 |
| CSTD | 0.8835 | 0.6499 | 0.8642 | 0.7742 |
These are not comparable to evaluations that supply the region count. Supplying it raises RWTD mIoU from 0.7843 to 0.8162.
Training
Two-stage curriculum on a single NVIDIA H100 NVL (95 GB). Stage 1 trains the Bridge,
mask head and dustbin with SAM 3 and Qwen frozen; Stage 2 unlocks LoRA on the Qwen
attention projections together with the masked [SEG] rows. The loss combines mask
cross-entropy, Dice, and a Shifted-Zero language objective that removes ordinary
linguistic pressure from the grounding state itself.
Licence
CC-BY-4.0 for these weights. The Qwen3-VL and SAM 3 base models keep their own licences, and this release does not relicense them.