AttnRes-50: 2D Attention Residuals for ResNet-50

Model Description

Try to extends the original 1D AttnRes (https://arxiv.org/abs/2603.15031) to 2D spatial feature maps for computer vision. However it seems not work on Classification :(

Dataset: ImageNet-100

ImageNet-100 is a 100-class subset of ImageNet-1K (ILSVRC2012). This study uses the split defined by Tian et al. (2020).

The train/val split is deterministic (torch.manual_seed(42) with randperm).

Training Details

Hyperparameters

Data Augmentation

• Training: RandomResizedCrop(224), RandomHorizontalFlip, ToTensor, Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])
• Validation: Resize(256), CenterCrop(224), ToTensor, Normalize (same stats)

Results

• Both attention variants underperform the standard ResNet-50 baseline on ImageNet-100 (~1–1.4% lower Acc@1).

Usage

import torch
from attnres_resnet_demo import AttnResNet50

# Load baseline (standard ResNet-50)
model = AttnResNet50(mode='none', num_classes=100)
ckpt = torch.load('result/best_none.pth', map_location='cpu')
model.load_state_dict(ckpt['model_state_dict'])
model.eval()

# Inference
from PIL import Image
import torchvision.transforms as T

transform = T.Compose([
    T.Resize(256),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
])

img = Image.open('example.jpg')
x = transform(img).unsqueeze(0)
with torch.no_grad():
    logits = model(x)
    pred = logits.argmax(dim=1)

Checkpoints

Each checkpoint contains:

{
    'epoch': int,
    'mode': str,            # 'none' | 'stage' | 'global'
    'model_state_dict': OrderedDict,
    'optimizer_state_dict': OrderedDict,
    'best_acc1': float,
    'num_classes': int,      # 100
}

Note for global mode: The model uses a lazy-initialized ModuleDict for cross-stage channel projections. When loading the checkpoint, run a dummy forward pass first to populate the projection layers before calling load_state_dict.

Limitations

• Small-scale study: ImageNet-100 is only 100 classes; conclusions may not transfer to full ImageNet-1K.
• No dual mode results: The dual-path variant was not trained due to negative findings from stage and global.
• Single training run per mode: No repeated runs with different seeds to measure variance.
• Standard training recipe: Uses a basic SGD + cosine annealing schedule without modern tricks (warmup, label smoothing, mixup, EMA).

Citation

If you use this code or findings, please cite the original AttnRes work:

@misc{chen2026attnres,
  title         = {Attention Residuals},
  author        = {Kimi Team  and Chen, Guangyu  and Zhang, Yu  and Su, Jianlin  and Xu, Weixin  and Pan, Siyuan  and Wang, Yaoyu  and Wang, Yucheng  and Chen, Guanduo  and Yin, Bohong  and Chen, Yutian  and Yan, Junjie  and Wei, Ming  and Zhang, Y.  and Meng, Fanqing  and Hong, Chao  and Xie, Xiaotong  and Liu, Shaowei  and Lu, Enzhe  and Tai, Yunpeng  and Chen, Yanru  and Men, Xin  and Guo, Haiqing  and Charles, Y.  and Lu, Haoyu  and Sui, Lin  and Zhu, Jinguo  and Zhou, Zaida  and He, Weiran  and Huang, Weixiao  and Xu, Xinran  and Wang, Yuzhi  and Lai, Guokun  and Du, Yulun  and Wu, Yuxin  and Yang, Zhilin  and Zhou, Xinyu},
  year          = {2026},
  archiveprefix = {arXiv},
  eprint        = {2603.15031},
  primaryclass  = {cs.CL}
}