Ember — the thermal sense for fusion-embedding-2

Ember is the first sense pack for fusion-embedding-2: it teaches the model to embed thermal infrared images in the same vector space as its text, image, video, and audio embeddings. Packs are named for the physical trace their sensor reads; Ember reads heat, and its sibling pack fusion-embedding-2-tactus reads touch (32x32 pressure/taxel arrays).

The family. Each sense is a separately loadable pack over the same frozen base: Tactus reads touch from a 32x32 pressure glove, Tactus Mat reads a 64x32 body pressure mat, Ember reads heat, and Tremor reads motion, with a Unitree-G1 head. Because the base is never modified, adding a sense costs a small trained head and an afternoon of compute rather than a new foundation model.

Ember is strictly additive. Technically it is a 44M-parameter gated adapter pack that attaches to the frozen decoder behind a thermal-only gate: when the gate is closed (every non-thermal input), the model's outputs are bit-for-bit identical to the model without the pack. This is verified, not aspirational; see Correctness below.

What it does

• Thermal image to text retrieval: R@10 0.785 on a held-out 2,000-caption gallery (frozen base: 0.224).
• Thermal zero-shot classification is preserved: LLVIP person/background 94.3 (calibrated ensemble harness; frozen base reference 95.4).
• Cross-domain thermal-to-visible retrieval: R@10 0.348 on LLVIP registered pairs, above the frozen baseline 0.165, without training on any LLVIP data.
• Text, RGB image, video, and audio embeddings unchanged, bit-for-bit.

Usage

Ember loads as an adapter pack through the multi-gate adapter registry in the fusion-embedding GitHub repository (fusion_embedding/adapters.py). Thermal images are single-channel; replicate to three channels and encode through the ordinary image path with the thermal scope open.

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from fusion_embedding.adapters import AdapterPacks
from inference import FusionEmbedder          # fe2_release/inference.py (GitHub repo)

emb = FusionEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-2b-preview", revision="v0.2-preview", device="cuda")

packs = AdapterPacks()
adapters, gate = packs.add_pack("thermal", emb.model.base_lm, 2048, rank=384)
adapters.load_state_dict(load_file(hf_hub_download(
    "EximiusLabs/fusion-embedding-2-ember", "model.safetensors")))
packs.to("cuda")

# thermal encode: thermal readout template, thermal scope open (the scope
# spans forward and backward)
import torch.nn.functional as F
text = ("<|im_start|>system\nRepresent this thermal infrared image.<|im_end|>\n"
        "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n"
        "<|im_start|>assistant\n")
inp = emb.proc(text=[text], images=[thermal_image_3ch], return_tensors="pt").to("cuda")
with torch.no_grad(), packs.scope("thermal"):
    h = emb.full(**inp).last_hidden_state
thermal_vec = F.normalize(h[0, inp["attention_mask"][0].sum() - 1].float(), dim=-1)

# everything else: leave the scope closed; outputs equal the pack-free model exactly
text_vec = emb.embed_text("a person crossing a dark road")
audio_vec = emb.embed_audio(wav, sr=16000)    # audio pack co-loaded, unaffected

The pack attaches equally to the raw base (AutoModel.from_pretrained("Qwen/Qwen3-VL-Embedding-2B"), attach at .language_model), which is the exact configuration it was trained in; both loading paths are verified bitwise in the release smoke. The readout is the standard fusion-embedding protocol: chat template, last non-pad token pooling, L2 normalization (see config.json for the exact templates and the trained temperature).

Evaluation

Training: contrastive thermal-to-caption alignment on IR-TD (61,320 pairs after FLIR exclusion and eval dedup), 3,900 steps, batch 16, 1,024 bank negatives, frozen base, bf16 base precision with fp32 adapters. Three seeds; seed 2 shipped.

Seed 2 holdout detail: R@1 0.412, R@5 0.692, R@10 0.785 over a 2,000-item gallery.

Text to thermal retrieval on the release holdout (queries shortened for display; retrieval used the full captions):

A caption-style ablation on the pre-exclusion corpus (82K pairs, 5,000 steps) found that caption richness is a generalization lever, not just an in-domain fit lever: training on full descriptive captions reached holdout R@10 0.843 and LLVIP twin 0.343, while first-sentence captions reached 0.594 and collapsed cross-domain transfer to 0.089, below the frozen baseline. Ember ships the full-caption arm.

Domain note: IR-TD spans 63 source collections but is still a finite domain mix. The LLVIP numbers above are cross-domain signal (night pedestrian scenes never seen in training), not a claim of parity with in-domain retrieval. Expect the gap to vary with distance from the training domains.

Correctness

The bit-for-bit preservation claim is tested at three levels:

1. Unit suite (GitHub repo, tests/test_thermal_adapters.py): closed-gate forwards equal the base exactly; gradients reach only the open pack; the gate must span forward and backward under gradient checkpointing.
2. Release checks with the trained weights: RGB-image and text forwards bit-for-bit equal to the pack-free base with the thermal gate closed, per seed.
3. Composability matrix (audio pack + thermal pack co-loaded on the same frozen decoder): audio through the registry vs the shipped single-gate path, audio co-loaded vs audio-only, thermal co-loaded vs thermal-only, and text / RGB / video vs the raw base, all bitwise; retrieval scores identical under co-load.

Mixed inputs that would open two gates in a single forward are outside the guarantee and are not tested.

Provenance

• Training corpus: IR-TD early access (IRGPT, ICCV 2025, arXiv:2507.14449, repository); 84,284 real thermal images with LLM-generated descriptive captions; academic research use only.
• FLIR exclusion: IR-TD includes FLIR-derived sources whose terms restrict redistribution of trained weights. All 20,964 images matching the FLIR capture signature (640x512) were excluded from training and holdout. This is a size-based heuristic, not an author-provided source mapping; the exclusion list ships in this repository (release_strip_640x512.json, sha256 65a870df5e4fe0008fd9bacd9fa81bd0c47f919779a28e398fea8d1bcabad09f).
• LLVIP is used for evaluation only (zero-shot gate and cross-domain retrieval).
• Eval hygiene: perceptual-hash dedup between the training set and the LLVIP test set found 0 collisions (hamming distance <= 4).

Deploy on RunPod

One-click from the RunPod Hub.

curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync \n  -H "Authorization: Bearer $RUNPOD_API_KEY" \n  -H "Content-Type: application/json" \n  -d '{"input": {"thermal": "<https url | data-uri | base64>"}}'

Use text instead of thermal to embed a caption. Returns 2048-d vectors, thermal and text in one space.

Engram

This pack is one of the modalities Engram searches. Engram is the open cross-modal memory layer for physical AI: it indexes a robot's video, audio, and motion into one embedding space and answers questions about it in plain language, including temporal reasoning that retrieval alone cannot.

pip install engram-robomem

Repo: https://github.com/Eximius-Labs/engram · PyPI: https://pypi.org/project/engram-robomem · Playground: https://www.eximiuslabs.com/playground

License

The Ember weights in this repository are released under CC-BY-NC-4.0 for research use, reflecting the academic-use terms of the training corpus. The core fusion-embedding-2 model is a separate artifact under its own license; this pack is optional and separable, and does not modify the core model's weights.