thermal-mae-vits-llvip-pilot

A ViT-S/16 MAE encoder for single-channel LWIR (thermal infrared) imagery, self-supervised-pretrained on 15,488 real thermal infrared frames from a public thermal and visible paired dataset. This is a training-mechanics pilot checkpoint, not a foundation model: read the honest caveats below before using it for anything.

What it is

• Architecture: ViT-S/16 encoder: embed dim 384, depth 12, heads 6, single input channel (Conv2d(1, 384, 16, 16) patch embed), 224x224 input, 21.4M encoder parameters (the full MAE model during training was 24.7M including the dim-256 depth-4 decoder, which is not shipped).
• Objective: masked autoencoding (MAE), mask ratio 0.75, per-patch z-score target normalization, activity-weighted masking.
• Data: the infrared half of a public thermal and visible paired dataset: 15,488 real LWIR night-street frames (12,025 train + 3,463 test JPEGs), resized to 256 in the corpus, random-resized-cropped to 224 for training. Every frame is a genuine thermal sensor capture, not a simulation. (See License and access below for the citation this dataset's terms require.)
• Training: 12,977 steps at batch size 256, AdamW (betas 0.9/0.95, weight decay 0.05), base LR 1.5e-4 with 5 percent warmup then cosine decay, drop path 0.1, bfloat16 autocast, 40.0 minutes wall clock on one RTX 5060 Ti 16 GB (peak VRAM 4,875 MB, sustained ~1,382 img/s).

Files

Load it against MAEViT from scripts/train_mae.py in the source repo:

import torch
sd = torch.load("encoder_real_v2.pt", map_location="cpu", weights_only=True)
model = MAEViT(image_size=224, patch_size=16, in_chans=1,
               embed_dim=384, depth=12, heads=6)
missing, unexpected = model.load_state_dict(sd, strict=False)
# missing = decoder weights only (not shipped); unexpected = []

Measured loss trajectory (verbatim, this checkpoint's run)

Masked-patch MSE on per-patch-normalized targets:

Mean of the first 20 logged points 0.976, mean of the last 20 0.655, minimum 0.6377 at step 11,040. Read honestly: the loss descended ~0.98 to ~0.65 (about a 33 percent reduction), with the bulk of the descent in the first ~5,000 steps and a flat noise-band coast once the cosine schedule hit its floor. That is a healthy MAE reconstruction curve on genuine thermal data, and it is the ONLY quality evidence this card claims.

Honest caveats (read before use)

1. This is a training-mechanics pilot, ~13x below the recipe's own floor. The source repo's PretrainRecipe.min_corpus_frames is 200,000; this run saw 15,488 frames. It demonstrates that the corpus builder, trainer, and MAE math work end to end on real LWIR sensor captures and produce a real descending loss. It makes no downstream quality claim: no detection, segmentation, or transfer benchmark was run on these weights. Nothing here says the learned features are good.
2. Commercial use is permitted under a separate provider permission, not the dataset's default public license. The dataset's own public license (per the authors' original release) grants use "for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation," and that remains the default for anyone downloading the public dataset directly. As of 2026-07, Dhi has obtained a separate, direct commercial-use permission from the dataset's provider covering this derivative checkpoint, so this checkpoint is no longer restricted to non-commercial use by upstream terms. Cite Jia et al., ICCV 2021 Workshops if you use it, regardless of use case.
3. 8-bit AGC'd JPEG input, not 16-bit radiometric. The source frames are camera-ISP-processed 8-bit JPEGs. The repo's radiometric framing ("absolute counts encode sensor identity") applies less cleanly to AGC'd data. Input normalization used the corpus's own measured stats (mean 74.67, std 47.40 on uint16-cast counts).
4. Corpus dedup was measured, then explicitly disabled for this run (--dedup-hamming -1): the repo's 8x8 average-hash dedup is built for continuous fixed-camera streams and measurably over-fires on this dataset's hand-curated night scenes (hamming<=4 kept only 4.8 percent of a 600-frame sample). All 15,488 frames were kept.

2026-07 evaluation redesign: two ViT-B/16 backbones, a corrected probe, and a corpus-contamination lesson

A later GPU-phase evaluation pass (2026-07-27 to 2026-07-28) trained and probed two additional, larger ViT-B/16 backbones (embed 768, depth 12, heads 12, a different, bigger architecture than the ViT-S/16 checkpoint whose weights are hosted in this repo above). Neither ViT-B/16 checkpoint's weights are published here or anywhere else yet. This section reports only their methodology and measured metrics, not a downloadable model. As of 2026-07, Dhi holds commercial-use permission from the relevant upstream providers (the public thermal and visible paired dataset, public multispectral pedestrian data, and the other A5 thermal sources), so licensing is no longer the reason these weights are withheld. They remain unpublished because they are early-stage research checkpoints, not yet validated for release: see the near-degenerate old probe and the unresolved backbone ranking discussed below.

The old crowding probe was near-degenerate

The crowding probe documented above (trained 0.6541 vs random-init 0.6359 vs majority 0.6526, on the 15,488-frame ViT-S/16 backbone) is a binary "at most one person vs two or more" task. Measured again on the two new ViT-B/16 backbones, the same near-degenerate pattern held:

A probe where random initialization lands within a few points of the majority baseline has almost no headroom left to show a trained backbone's actual quality: a "win" of a point or two is not distinguishable from noise on that evidence alone.

The replacement: a 4-way person-count probe with real headroom

The replacement probe buckets images into 0 / 1 / 2-3 / 4+ people, built from real COCO-format person-box annotations (a Roboflow export of a public thermal imagery dataset's free-sample thermal images, CC BY 4.0, commercially clean for anyone, unlike the paired thermal dataset's default public terms (non-commercial-only for the general public; Dhi separately holds commercial permission for its own derivatives of that paired dataset, see License and access below). Its majority-class baseline measures 0.3237 (never above 0.5) versus the old task's 0.6526, giving real headroom to actually separate a good backbone from a bad one.

Before being trusted on real backbones, the replacement probe validated itself against two synthetic sanity checks: on features literally derived from the labels plus small noise (known-informative), it reported 1.0 accuracy with an unambiguous trained-beats-random win; on pure noise features for both "trained" and "random" arms, it reported no false win (trained 0.26 vs random 0.245 accuracy, delta 0.015), correctly refusing to manufacture a signal that is not there. Only after passing both checks was the probe run on the real backbones below.

Measured results (frozen-feature ridge probe, same protocol both backbones), each row read directly from that backbone's own result file:

Both backbones beat their own random-init control and the 0.3237 majority baseline. production_accepted is false for both; this is research evidence, not a production gate.

The ranking between these two backbones is not resolved. The two random-init controls differ by 0.107 (0.5379 vs 0.4311) purely because they are two separate random draws, not a shared control, while each backbone's own improvement over its own control is nearly identical (0.0758 vs 0.0847). That means the apparent absolute gap between the two backbones is confounded by initialization variance, not a clean signal that the smaller or larger corpus produced a better backbone. Read this as "both beat random init and majority," not as "backbone A beats backbone B."

The corpus-contamination finding

An earlier version of the 47,706-frame backbone's corpus (a 102,254-frame draw, before the fix below) scored a frozen-probe accuracy of 0.4791 on the old crowding task, below both random-init (0.652) and the majority-class baseline (0.6526), despite training loss falling normally the entire run (0.94 down to 0.4761). Investigation traced the cause to unrestricted file globs in the corpus-fetching script pulling both modalities out of paired visible-plus-thermal datasets: exactly 11,615 of a public multispectral pedestrian dataset's 23,178 files (50.1 percent) turned out to be its visible/ (ordinary color-camera RGB) folder, not its lwir/ folder, and a separate source (a public thermal person-detection dataset) turned out to be majority false-color "ironbow" visualizations that cannot be turned back into a real radiometric signal by a naive luma reduction. A large fraction of the nominally "thermal" corpus was ordinary photography or a scrambled color-palette proxy, trained under one shared masked-autoencoding objective: the loss curve looked completely healthy the whole time (reconstructing photo texture is easy) while the encoder's transfer to genuine thermal frames collapsed below random initialization. Only the random-init comparison caught it; the loss curve alone did not. The fix restricted the multispectral-pedestrian and paired-thermal-dataset globs to their lwir//infrared/ subfolders and added a rejection filter for any 3-channel frame whose raw channels diverge past a measured threshold; retraining on the resulting cleaned 47,706-frame corpus produced the ckpt_a5_v2 backbone reported in the table above.

Files behind this section

• probe_labeled_old15k.json, probe_labeled_new47k.json (frozen 4-way probe results, one file per backbone).
• probe_a5_result.json, probe_a5v2_result.json (the old crowding-probe results, showing the near-degenerate pattern and the contaminated-corpus regression).
• tests/test_probe_mae_labeled.py in the source repo (the probe's own self-validation tests).
• scripts/train_mae_real.py's module docstring in the source repo (the contamination investigation, written up in full at the time it was found and fixed).

Provenance

• Repo: DHI-Technologies-Inc/thermal-perception, commit bcc1ef2ffbb41f0688a70a758edb3b51bf8d4b2a ("feat(ssl): MAE pretraining pilot on REAL thermal data (paired thermal/visible dataset, infrared half) (#14)") plus an --encoder-out / --config-out checkpoint-saving addition to scripts/train_mae_real.py.
• Command:

  CUDA_VISIBLE_DEVICES=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
    python scripts/train_mae_real.py \
    --image-dir <paired-thermal-dataset>/infrared --glob "*/*.jpg" \
    --corpus-root ~/a5real-work/corpus --dedup-hamming -1 --store-size 256 \
    --batch-size 256 --base-lr 1.5e-4 --weight-decay 0.05 --drop-path 0.1 \
    --mask-ratio 0.75 --max-minutes 40 --warmup-frac 0.05 \
    --encoder-out encoder_real_v2.pt --config-out run_config_real_v2.json
  

• Date: 2026-07-10, single RTX 5060 Ti 16 GB (vast.ai), torch 2.12.0+cu130, seed 0.
• Note on regeneration: the originally documented pilot run (see REAL_DATA_RESULTS.md in the source repo: 12,974 steps, loss ~0.98 to ~0.65, min 0.6351 at step 11,040) did not save weights. This checkpoint is a fresh run of the identical command, data, and seed on the same GPU; it reproduced the documented run within noise (12,977 steps, mean of last 20 points 0.655 vs 0.653, min 0.6377 vs 0.6351 at the same step 11,040, identical corpus stats and peak VRAM). The loss table above is from THIS run, the one that produced the shipped weights.
• Numerical sanity gate: sanity_check_against_numpy() passed with loss diff 0.00e+00 before training started.

Links

• Thermal benchmark dataset: Dhi-Technologies/thermal-perception-benchmark
• Live demo Space: Dhi-Technologies/thermal-perception-demo

License and access

Released under other (a bespoke commercial-permission derivative license, not a standard SPDX identifier such as MIT or Apache-2.0, since Dhi has not granted one of those). Dhi has obtained a separate, direct commercial-use permission from the dataset's provider covering this derivative checkpoint, so this checkpoint is not restricted to non-commercial use by the dataset's upstream terms. The dataset's own public license remains non-commercial-only for anyone using the public dataset directly. Citation required by that dataset's terms: cite Jia et al., "LLVIP: A Visible-infrared Paired Dataset for Low-light Vision" (ICCV 2021 Workshops), in any publication or output that uses this checkpoint. Dhi's permission covers this specific derivative only, not a general re-licensing of the dataset itself. Access is gated as Dhi's own access-control preference, not a licensing restriction; redistribution of this checkpoint without Dhi's permission is still not allowed. Commercial licensing (Dhi's own pricing) via dhi-tech.com.

Commercial licensing

Research and evaluation use is free. Production and commercial use is licensed self-serve with published prices.

• Get a license: https://dhi-tech.com/buy/thermal-perception
• All published prices: https://dhi-tech.com/pricing/