ResNet-18 + LSTM Radiology Report Generator
This repository contains a lightweight, clinically optimized ResNet + LSTM autoregressive image-to-text model designed for generating textual descriptors from Chest X-ray (CXR) images. The architecture is engineered specifically for scenarios with constrained computational resources and exhibits high semantic alignment on specialized medical datasets.
Model Details
- Developed by: Biomedical Image Analysis Laboratory (Lab225), United Institute of Informatics Problems, National Academy of Sciences of Belarus (UIIP NASB), Minsk.
- Architecture: Fine-tuned ResNet-18 Image Encoder (fully unfrozen in the final stage) paired with a Deep 5-layer LSTM Decoder.
- Parameters: ~20 Million parameters.
- Vocabulary: Word-level tokenization containing 2,190 unique tokens (including punctuation and special
<BOS>,<EOS>,<PAD>,<UNK>tokens), initialized via pre-trained GloVe (Global Vectors for Word Representation) embeddings. - Primary Task: Automated generation of radiology findings and textual chest X-ray descriptors.
Inference Widget
You can test this model directly in the browser via the Hugging Face Inference Widget on the right. Upload a standard chest X-ray image (PNG/JPEG) to evaluate the generated output sequence.
Performance Analysis & Metrics (Evaluation)
In comprehensive benchmark trials on the Belarusian screening dataset, this lightweight architecture was evaluated against heavily parameterized modern models, including VisionGPT2 (210M parameters) and a fine-tuned Qwen3.5-4B M-LLM.
Critical Scientific Insight
ResNet-18 + LSTM (configured with top_p=0.1 decoding) dominates foundational semantic alignments.
Despite its exceptionally low parameter footprint compared to massive transformer pipelines,
its focused, medical-domain word-level dictionary space allows it to extract and sequence
localized anatomical findings with superior stability.
Quantitative Test Set Metrics
Evaluation results highlighting the top performing clinical metrics for the leading radiology report generation (RRG) evaluation frameworks:
| Model Configuration | GREEN | RadCliQ_Inv | RaTEScore | RadGraph_F1 |
|---|---|---|---|---|
ResNet-18 + LSTM (top_p=0.1) | 0.264 | 0.315 | 0.561 | 0.224 |
Data Pipeline & Preprocessing
- Report Processing: ~211,000 raw Russian radiology records underwent semantic/linguistic corrections via asynchronous LLM APIs (removing administrative metadata, acquiring OCR artifacts, expanding abbreviations) and strict translation to English using a Fleischner Society Glossary-compliant framework.
- Dataset Filtering: A highly curated subset of 46,599 unique image-text pairs was derived by filtering for clinical target keywords (
tuberculosis,pneumonia,scoliosis,fibrosis,nodules,emphysema,pneumosclerosis,sclerosis). - Data Split: 80% Train (37,279), 10% Validation (4,660), 10% Test (4,660).
- Training Schedule: Phase 1: 50 epochs with a frozen ResNet encoder. Phase 2: Extended training up to 240 epochs with a deep 5-layer LSTM decoder. Phase 3: Unfreezing the ResNet-18 image encoder at a lower learning rate (
3e-6) using a Cosine Annealing scheduler.
Quick Start (How to Use)
To load this custom architecture natively via the Hugging Face transformers library,
ensure you have configuration_resnet_lstm.py and modeling_resnet_lstm.py
in your local directory or let the hub download them automatically:
from transformers import AutoModel
import torch
# Load the custom Lab225 architecture directly from the hub
model = AutoModel.from_pretrained("lab225/resnet-lstm-belarus-screening", trust_remote_code=True)