SkyVLM
A 99.4M-parameter vision-language model for satellite imagery, trained from scratch on a single 8 GB laptop GPU.
It writes a structured caption such as
a satellite image of <main object>, surrounded by <object>; <object>.
Code, training scripts, a Streamlit demo and full results: https://github.com/thorOdinson16/SkyVLM
Model
image ─► ViT (25.7M; masked-image-modeling pretrained, then CLIP-aligned)
└─ 196 patch tokens ─► 2-layer MLP projector (1.05M) ─► LM (72.6M decoder) ─► caption
| Component | Parameters |
|---|---|
| Vision encoder (ViT, 8 layers, 512 dim, 16×16 patches, 224 px) | 25.7M |
| MLP projector (512→1024→512) | 1.05M |
| Language model (20 layers, RoPE, RMSNorm, 16k vocabulary) | 72.6M |
| Total | 99.4M |
The language model was pretrained on ~1B tokens (90% FineWeb-Edu, 10% SkyScript captions), the encoder on 200k unlabelled SkyScript images and then aligned to captions with a CLIP-style loss, and the whole model was fine-tuned on 200k SkyScript image-caption pairs (projector first, then everything).
Usage
This is a custom PyTorch architecture, not a transformers model. Download the repo and run:
pip install torch torchvision safetensors sentencepiece pandas pillow
python inference.py my_image1.jpg my_image2.png
or from Python:
from PIL import Image
from inference import load, caption
model, tok, cfg = load(".") # repo directory
print(caption(model, tok, cfg, [Image.open("my_image.jpg")])[0])
Images are resized to 224×224 and decoded greedily. The output follows the SkyScript caption template (OpenStreetMap-style tags).
Results
Official SkyScript test captions, all 30k images, greedy decoding:
| Model | CIDEr-D | BLEU-4 | ROUGE-L | Main object exact | Surrounding-object F1 |
|---|---|---|---|---|---|
| SkyVLM (this model) | 1.90 | 0.475 | 0.643 | 21.3% | 0.386 |
| Same recipe without CLIP alignment | 1.75 | 0.464 | 0.632 | 19.6% | 0.372 |
| Same recipe, random-init ViT (first 5k test images) | 0.69 | 0.323 | 0.530 | 5.9% | 0.264 |
Shuffled or blank images drop CIDEr-D to 0.09 / 0.04, so the model uses the image.
Image-text retrieval on 1,000 test pairs, ranking captions by log p(caption | image) (image→text uses a PMI correction):
| Model | Image→text R@1 | Text→image R@1 |
|---|---|---|
| SkyVLM | 36.5 | 34.7 |
| SkyCLIP ViT-B/32 (external reference) | 8.6 | 6.8 |
SkyCLIP is a general remote-sensing model while SkyVLM is a specialist trained on these templated captions, so this is a reference point rather than a claim of a better model.
Vision encoder
RESISC45 linear probe (45 scene classes, 6,300 test images, frozen features; no RESISC45 data seen in training):
| Encoder | Test accuracy | Macro-F1 |
|---|---|---|
| Random ViT (untrained) | 43.5% | 0.425 |
| MIM-pretrained (100 epochs, 200k unlabelled images) | 78.6% | 0.785 |
| MIM + CLIP alignment (this model's encoder) | 88.4% | 0.885 |
| ImageNet ViT-S/16 (supervised, ~14M labelled images) | 88.5% | 0.885 |
Language model
FineWeb-Edu validation perplexity 21.7 (72.6M parameters, 983M training tokens). SkyScript caption validation loss bottomed at 0.76 and ended at 0.94 as the model began to memorize the captions (10% of each batch, ~18 passes).
Sanity checks
Main-object category (building, road, railway, ...) is right 61.5% of the time versus 21% for always guessing the most common one (measured on the variant without CLIP alignment); all outputs follow the caption template.
Single training run per condition; no variance estimates.
Limitations
- Trained only on SkyScript's templated, OpenStreetMap-derived captions: it will not follow free-form prompts or answer questions.
- Typical errors are the right category with the wrong specifics (an apartment block described as an office). Exact main-object match is 21%, and references have several valid descriptions per image, so n-gram and exact-match scores understate quality.
- Images are ~200 px at 224×224 input; fine detail is limited.
- Not evaluated outside SkyScript imagery, on other regions' sensors, or for any safety-critical use.
Data and license
Apache-2.0. Trained on SkyScript (MIT), which pairs Google Earth Engine imagery with OpenStreetMap-derived captions. OpenStreetMap data is © OpenStreetMap contributors (ODbL); imagery remains subject to its providers' terms. FineWeb-Edu (ODC-By) was used for language-model pretraining.