Turkish CLIP (ViT-B/32 + ModernBERT-TR)
CLIP-style dual encoder that maps images and Turkish text into a shared 512-dimensional embedding space. Supports image → Turkish text and Turkish text → image retrieval.
Architecture
| Component | Details |
|---|---|
| Image encoder | OpenAI CLIP ViT-B/32 vision tower (openai/clip-vit-base-patch32), 12 layers, patch 32, 224×224 input; representation = pooler_output (post-LayerNorm CLS, 768-d) |
| Text encoder | ytu-ce-cosmos/modernbert-tr-base (ModernBertModel, 22 layers, hidden 768, no MLM head) |
| Text pooling | Masked mean pooling over last_hidden_state using attention_mask (no CLS pooling) |
| Image projection | Linear(768 → 512), initialised from CLIP's pretrained visual_projection (bias = 0) |
| Text projection | Linear(768 → 512), random init |
| Normalisation | L2 on both embeddings; similarity = cosine |
| Temperature | Learnable logit_scale, init log(1/0.07), exp(logit_scale) clamped to ≤ 100.0; final value 19.63 |
| Objective | Symmetric CLIP InfoNCE: (CE(logits_per_image, arange(B)) + CE(logits_per_text, arange(B))) / 2 |
| Parameters | 87.5M (vision) + 148.7M (text) + 0.79M (projections) + 1 (logit_scale) = 237.0M |
image → ViT-B/32 → 768 → Linear → 512 ─┐
├→ L2 norm → cosine × exp(logit_scale) → symmetric InfoNCE
text → ModernBERT-TR → masked mean → 768 → Linear → 512 ─┘
Preprocessing
- Image:
CLIPImageProcessorofopenai/clip-vit-base-patch32— RGB, resize shortest edge to 224 (bicubic), center crop 224, rescale 1/255, normalise with mean(0.48145466, 0.4578275, 0.40821073), std(0.26862954, 0.26130258, 0.27577711). - Text: ModernBERT-TR tokenizer, cased (no lower-casing),
[CLS] … [SEP], truncation atmax_length=128, dynamic padding.
Training data
atasoglu/flickr30k-turkish: Turkish translations of Flickr30k captions, 5 captions per image. Official Karpathy image-level splits shipped with the dataset:
| Split | Images | Captions |
|---|---|---|
| train | 29,000 | 145,000 |
| validation | 1,014 | 5,070 |
| test | 1,000 | 5,000 |
No image appears in more than one split (verified). During training each image is sampled once per epoch with one randomly chosen caption (distinct captions per image; 255 within-image duplicates removed), so a batch never contains two captions of the same image.
Caption length (ModernBERT-TR tokens incl. special tokens, all splits): min 4, mean 14.6, median 14, p90 21, p95 24, p99 32, max 200.
Training procedure
| Setting | Value |
|---|---|
| Frozen | image encoder: true, text encoder: false |
| Epochs | 15 (selected checkpoint: epoch 15, step 6795) |
| Batch size | 64 (gradient accumulation 1; in-batch negatives per step = batch size − 1) |
| Optimizer | AdamW, betas (0.9, 0.98), eps 1e-06, weight decay 0.01 (not on biases / norms / logit_scale) |
| Learning rates | text encoder 1e-05, projections 1e-04, logit scale 1e-04, image encoder 1e-06 (only if unfrozen) |
| Schedule | linear warmup (0.1 of steps) + cosine decay to 0.0 × peak |
| Precision | autocast bfloat16; loss and logits in float32 |
| Gradient clipping | max norm 1.0 |
| Seed | 42 |
| Checkpoint selection | best validation mean_recall |
Evaluation
Retrieval over the full split. Image → text: a query image is correct at K if any of its 5 captions is in the top K. Text → image: a caption is correct at K if its image is in the top K.
Test (evaluated once, after checkpoint selection)
| Direction | R@1 | R@5 | R@10 |
|---|---|---|---|
| Image → Text | 68.30 | 91.70 | 95.70 |
| Text → Image | 54.38 | 83.04 | 90.18 |
Mean recall: 80.55 · RSUM: 483.30 · 1000 images / 5000 captions
Validation (selected checkpoint)
| Direction | R@1 | R@5 | R@10 |
|---|---|---|---|
| Image → Text | 71.99 | 92.90 | 96.25 |
| Text → Image | 54.99 | 83.33 | 89.98 |
Mean recall: 81.57 · RSUM: 489.45 · 1014 images / 5070 captions
Usage
Files: model.safetensors (all weights), config.yaml, modeling_turkish_clip.py, image_encoder/ (vision config + processor), text_encoder/ (ModernBERT-TR config + tokenizer). Requires torch, transformers>=4.48, safetensors, pillow, pyyaml.
import torch
from PIL import Image
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("thealper2/turkish-clip-vit-b32-modernbert-tr")
sys.path.insert(0, path)
from modeling_turkish_clip import TurkishCLIP
model = TurkishCLIP.from_pretrained(path, device="cuda" if torch.cuda.is_available() else "cpu")
with torch.no_grad():
image_embedding = model.encode_image(Image.open("photo.jpg")) # [1, 512], L2-normalised
text_embedding = model.encode_text("Bir parkta oynayan çocuk.") # [1, 512], L2-normalised
similarity = image_embedding @ text_embedding.T # cosine similarity
captions = ["Karda koşan iki köpek.", "Sahnede gitar çalan bir adam.", "Bir parkta oynayan çocuk."]
scores = model.encode_image(Image.open("photo.jpg")) @ model.encode_text(captions).T
print(scores.softmax(dim=-1))
encode_text accepts a string or a list of strings (or input_ids + attention_mask); encode_image accepts a PIL image, a list of PIL images, or preprocessed pixel_values.
Limitations
- Trained only on Flickr30k-domain images (people, animals, everyday scenes); 29k images is small for contrastive pre-training, so out-of-domain generalisation is limited.
- Captions are machine translations of the English Flickr30k captions and contain translation artefacts (e.g. a few degenerate repeated captions up to 200 tokens).
- The image encoder was frozen; image features are those of the original OpenAI CLIP ViT-B/32.
- Identical captions attached to different images (e.g. "İki köpek karda oynuyor.") can appear in the same batch and act as false negatives.
Licenses
- OpenAI CLIP ViT-B/32 weights: MIT (openai/CLIP).
ytu-ce-cosmos/modernbert-tr-base: Apache-2.0.- Flickr30k images and captions: Flickr30k terms of use (non-commercial research);
atasoglu/flickr30k-turkishis markedlicense: other.