IMG — Relational Pattern-Based Similarity Metric

A Universal Similarity Metric for Computer Vision

Author: Imam Ghozali — Independent Researcher 📧 imam.gh98@gmail.com

Model Description

Traditional similarity metrics such as cosine similarity compare embedding vectors through global angular relationships.

IMG introduces a different paradigm: instead of comparing absolute vector values, IMG compares local relational patterns inside the embedding.

The proposed framework consists of three complementary metrics:

1. IMG Sign Score
2. AMP IMG Score
3. Chain Score

Note: This work does not propose replacing cosine similarity. Instead, IMG is proposed as an alternative similarity metric. Experimental results suggest that the optimal similarity metric depends on how the embedding itself is learned.

Relational Learning Hypothesis

In Javanese, one expresses gratitude as "matur suwun"; in Sundanese, the same sentiment is conveyed as "hatur nuhun". Despite different surface structures, both phrases encode identical meaning through internally consistent relational patterns.

This linguistic observation inspired the central hypothesis of this work:

Identity can be encoded through consistent relational patterns rather than absolute values.

Instead of forcing embeddings to occupy a specific angular position, the proposed method trains the network to preserve local relational consistency. Similarity is then evaluated by comparing relational patterns rather than absolute vector orientation.

Relational Training Objective

Unlike ArcFace, which explicitly optimizes cosine similarity using Angular Margin Loss:

$$ L_{ArcFace} = -\log \frac{e^{s\cos(\theta_y+m)}}{e^{s\cos(\theta_y+m)}+\sum_j e^{s\cos\theta_j}} $$

the proposed method directly optimizes the desired similarity metric itself.

For two embeddings $E_1, E_2 \in \mathbb{R}^{1024}$, the objective is to maximize their local sign agreement.

Soft Sign Agreement

For each embedding dimension:

$$ a_i = \frac{\tanh(\beta E_{1,i} E_{2,i}) + 1}{2} $$

where a positive product indicates agreement and a negative product indicates disagreement. Unlike a hard sign comparison, the hyperbolic tangent provides a smooth, differentiable approximation.

Sliding Window Aggregation

For each sliding window:

$$ S_k = \sum_{i=k}^{k+W-1} a_i $$

with window size $W = 11$ and threshold $T = 8$.

Differentiable Matching Gate

$$ M_k = \sigma\big(50(S_k - T + 0.5)\big) $$

which approximates:

$$ M_k \approx \begin{cases} 1 & \text{if } S_k \ge T \ 0 & \text{if } S_k < T \end{cases} $$

while remaining differentiable.

IMG Sign Score

$$ IMG(E_1, E_2) = \frac{1}{N}\sum_{k=1}^{N} M_k, \qquad N = d - W + 1 $$

Relational Loss

For positive pairs:

$$ L_{same} = (1 - IMG)^2 $$

For negative pairs:

$$ L_{diff} = IMG^2 $$

Final objective:

$$ L = L_{same} + L_{diff} $$

This is exactly the objective used during training — no angular-margin loss, cosine loss, or triplet loss is involved.

Training Data & Hyperparameters

Positive pairs consist of two images belonging to the same identity, while negative pairs are randomly sampled from different identities.

Why Does This Matter?

Traditional face-recognition losses optimize embeddings for cosine similarity. The proposed approach instead optimizes embeddings directly for the intended inference metric. Consequently:

• Embeddings trained with Angular Margin Loss naturally favor cosine similarity.
• Embeddings trained with the proposed relational loss naturally favor IMG Sign.

This suggests that the similarity metric and the embedding loss should be designed together rather than independently.

Key Idea

Model Architecture

SW357 Block

Conv2 → Conv3 → Conv4 → Conv5 → Conv6 → Conv7 → Conv8 → Conv9 → Conv10
     → Global Average Pooling → FC → BatchNorm

Evaluation Results

SW357 Embedding (native)

ArcFace Evaluation (relational metric tested on external embedding)

Observation: Cosine remains the best metric for ArcFace because ArcFace is explicitly optimized using Angular Margin Loss. However, IMG Sign remains highly competitive despite never being used during ArcFace training.

Main Finding

Results suggest that Similarity Metric and Embedding Loss Function should be considered together:

• Embeddings trained with Angular Margin Loss naturally favor cosine similarity.
• Embeddings trained with the proposed relational loss naturally favor IMG Sign.

Therefore, there is no universally best similarity metric. The optimal metric depends on how the embedding space is learned.

Domain-Agnostic Potential (Beyond Computer Vision)

While evaluated on face verification, the core mathematics of the IMG Framework are inherently domain-agnostic. Because it discards absolute magnitude dependency and focuses entirely on local sign-pattern agreements, this framework can be generalized to non-visual embeddings:

• Audio & Speech Processing: By applying IMG to audio spectrogram embeddings, the metric can eliminate amplitude/volume variations (gain changes), establishing a noise-robust framework for voice biometrics.
• Structural Bioinformatics: In protein structural analysis, exact physical distances fluctuate due to environment/simulations. IMG can be applied to capture invariant relational topology patterns between amino acids rather than relying on strict absolute spatial coordinates.

Metric Reference Implementations

IMG Sign Score

def img_sign_score_np(e1, e2):
    n = len(e1) - WINDOW_SIZE + 1
    mc = 0
    for i in range(n):
        s1 = np.where(e1[i:i+WINDOW_SIZE] >= 0, 1, -1)
        s2 = np.where(e2[i:i+WINDOW_SIZE] >= 0, 1, -1)
        if np.sum(s1 == s2) >= THRESHOLD:
            mc += 1
    return mc / n

AMP IMG Score

def amp_img_score_np(e1, e2):
    n = len(e1) - WINDOW_SIZE + 1
    total = 0
    for i in range(n):
        w1 = e1[i:i+WINDOW_SIZE]
        w2 = e2[i:i+WINDOW_SIZE]
        s1 = np.where(w1 >= 0, 1, -1)
        s2 = np.where(w2 >= 0, 1, -1)
        if np.sum(s1 == s2) >= THRESHOLD:
            a1 = np.mean(np.abs(w1))
            a2 = np.mean(np.abs(w2))
            total += max(0, 1 - abs(a1 - a2) / max(a1, a2, 1e-6))
    return total / n

Chain Score

def chain_score_np(e1, e2):
    n = len(e1) - WINDOW_SIZE + 1
    flags = []
    for i in range(n):
        ...
    total = sum(flags)
    img_sign = total / n
    ...
    avg_chain = total / n_chains
    diff = avg_chain - NEUTRAL_LEN
    score = img_sign + (
        REWARD_RATE * diff
        if diff >= 0
        else PUNISH_RATE * diff
    ) / 100
    return np.clip(score, 0, 1)

Datasets

train/
  train_sw357_conv10_imgsign_a100.py  — Training on A100/Colab
  train_eval_sw357_conv10_gtx.py      — 1-epoch test on GTX
  train_eval_sw357_conv13_gtx.py      — Conv13 variant test
  precrop_casia.py                    — Pre-crop CASIA with MTCNN

eval/
  eval_lfw_gtx_chain_conv10.py        — Eval Conv10 + Chain Score (GTX)
  eval_lfw_gtx_imgsign_conv10.py      — Eval Conv10 IMG Sign (GTX)
  eval_benchmarks_a100.py             — Multi-dataset benchmark (A100)
  eval_metric_comparison_a100.py      — FaceNet/ArcFace metric test

app/
  face_compare_conv10.py              — Desktop UI comparison app (tkinter)

How to Use

1. Install dependencies

pip install torch torchvision facenet-pytorch insightface Pillow numpy scikit-learn

2. Download checkpoint

Place best_model_epoch39_plateau.pth in your working directory.

Mirror download link: https://zenodo.org/records/21232756

3. Eval on LFW

# Edit CKPT_PATH and LFW_DIR in the script first
python eval_lfw_gtx_imgsign_conv10.py

4. Run comparison app

python face_compare_conv10.py

Voting System

Three metrics, one threshold (from IMG Sign sweep):

2/3 or 3/3 pass → ✅ MATCH
1/3 pass        → ⚠️  UNCERTAIN
0/3 pass        → ❌ DIFFERENT

During development of the interactive ablation visualizer, a preliminary observation was made using the custom polygon occlusion tool:

Observation: Region-specific embedding sensitivity

When occluding specific facial regions (e.g., right eye) using a custom polygon mask and comparing the resulting embedding changes across two different individuals:

Same person, different photos: occluding the same region produces delta spikes at similar embedding dimensions across both photos Different people: occluding the same region produces delta spikes at different embedding dimensions, or in some cases near-zero delta for one person (e.g., when glasses obscure the region)

Example (custom polygon, right eye region):

Same person: Photo 1: changed 4/1014 windows = 0.4% spike_delta: 110 Photo 2: changed 1/1014 windows = 0.1% spike_delta: 100 → spike locations visually correlated

Different people: Photo 1: changed 4/1014 windows = 0.4% spike_delta: 110 Photo 2: changed 0/1014 windows = 0.0% spike_delta: 97 → spike locations differ significantly

This observation suggests that the IMG Sign MSE loss function, through its overlapping sliding window structure, may induce implicit spatial organization in the embedding space — where different facial regions influence different embedding dimensions. However, this has not yet been formally tested and should be treated as a preliminary observation pending rigorous evaluation.

⚠️ This is an informal observation from the visualization tool, not a validated experimental result. Formal ablation study with multiple identities and statistical analysis is planned as future work.

Conclusion

IMG is proposed as an alternative similarity metric rather than a replacement for cosine similarity. Experiments indicate that cosine similarity performs best for embeddings trained with angular-margin objectives, while IMG Sign performs best for embeddings trained with the proposed relational objective. The framework is model-agnostic and can be applied to embeddings generated by different architectures.

Citation

If you use this work, please cite via:

• Zenodo (DOI): https://doi.org/10.5281/zenodo.21232755
• GitHub: https://github.com/imamgh11/imgnet
• Hugging Face: https://huggingface.co/imghost11/imgnetV1

License

This project is licensed under the MIT License.