chinese-clip-ft-sinonom
Code, data setup and full reproduction steps:
https://github.com/vo-hoang-kh4ng/comparing-character-glyph-images-Sino-N-m
(the README.md there walks the whole pipeline, with the expected value to check after each step;
the write-up is report/paper.pdf.)
Chinese-CLIP ViT-B/16 image tower fine-tuned to retrieve Sino-Nôm (Hán-Nôm) glyph images across typefaces. The training objective is typeface invariance: the same Unicode codepoint rendered in different fonts should map to the same vector.
Built for a course project on similarity search over a 26,044-glyph Sino-Nôm corpus, where the query is a handwritten scan and the corpus is printed regular script — a distribution gap that zero-shot models handle poorly.
Results
Measured on 59 real handwritten scans whose labels are decoded from filenames and checked by a human reader. No pixel of the evaluation set entered training (enforced by an assertion, not by convention).
Whole-corpus retrieval over all 26,044 characters:
| model | hit@1 | hit@10 | hit@20 | MRR | 95% CI of hit@1 |
|---|---|---|---|---|---|
chinese-clip zero-shot (this base) | 0.0169 | 0.0678 | — | 0.0327 | — |
chinese-clip-large zero-shot | 0.0847 | 0.1864 | 0.2034 | 0.1143 | [0.028, 0.187] |
| this model | 0.5932 | 0.7966 | 0.8644 | 0.6691 | [0.457, 0.719] |
The two confidence intervals are disjoint. Random baseline is 4.7e-4, so hit@1 is ~1250x chance.
Ranking within a same-reading candidate group (median 7 candidates): top-1 accuracy 0.8136 versus 0.4746 for the zero-shot large model and 0.4407 for a colour-histogram baseline.
Not memorisation
2,604 Unicode classes were held out by class — every view removed, not a random image split.
| trained classes | held-out classes | |
|---|---|---|
| before fine-tuning | 0.7938 | 0.8041 |
| after fine-tuning | 0.9954 | 0.9939 |
Equal. SupCon has no learnable class centroids, so there is nothing to memorise; what transfers is a comparison function, and it carries over to characters never seen.
Usage
The architecture and embedding width are unchanged from the base model, so the state dict loads
into ChineseCLIPModel directly.
import torch
from transformers import AutoImageProcessor, ChineseCLIPModel
from huggingface_hub import hf_hub_download
REPO = "vohoangkh4ng/chinese-clip-ft-sinonom"
model = ChineseCLIPModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
state = torch.load(hf_hub_download(REPO, "best_fp16.pt"), map_location="cpu")["model"]
missing, _ = model.load_state_dict(state, strict=False)
assert not [k for k in missing if not k.startswith(("text_model", "text_projection"))]
processor = AutoImageProcessor.from_pretrained(
"OFA-Sys/chinese-clip-vit-base-patch16", use_fast=False)
model = model.half().eval()
inputs = processor(images=[img], return_tensors="pt") # img: PIL, RGB
with torch.no_grad():
v = model.get_image_features(pixel_values=inputs["pixel_values"].half())
v = torch.nn.functional.normalize(v.float(), dim=1) # cosine == dot product
strict=False is required because the text tower is dropped; the assertion above still fails loudly
if any vision key is missing, so a partial load cannot pass silently.
Weights are stored in fp16. This is not an approximation — the inference path casts to fp16 right after loading, so storing fp16 only performs a cast that always happened. Verified by rebuilding all 26,044 corpus vectors from both precisions: maximum absolute difference 0.0.
Training
| objective | SupCon (supervised InfoNCE), temperature 0.07 |
| labels | Unicode codepoint only |
| sampling | P×K: 24 classes × 2 views per batch |
| data | 141,981 images across 23,440 classes, rendered through 7 Hán-Nôm fonts |
| augmentation | scan-degradation applied online, p = 0.6, strength 1.0 |
| optimiser | AdamW, lr 1e-5, 300-step warmup, cosine, bf16, gradient checkpointing |
| schedule | 6 epochs, best at epoch 5; ~4 min/epoch, 3.7 GB VRAM |
Labels are deliberately only the codepoint — not radical or stroke count. Training on a linguistic label and then reporting that label's metric measures the training set rather than retrieval quality.
The degradation model is calibrated against measurements of real scans (edge, contrast, stroke thickness, fill, noise, background) rather than tuned by eye.
P×K sampling is required, not a refinement. A random batch of 48 drawn from 23,440 classes contains ~0.05 positive pairs in expectation, so nearly every batch would have nothing to pull together.
Limitations
- One run, one seed. Per-epoch hit@1 varied 0.51–0.59, which is the size of the measurement noise, so read 0.5932 as "around 0.55".
- n = 59 is the whole real-scan evaluation set. Only the zero-shot-vs-fine-tuned gap is large enough to clear that interval; smaller comparisons are not resolvable.
- Backbone is ViT-B/16, not large — a VRAM constraint during training, not a modelling choice.
Fine-tuning
largeis untested. - Rendered images cannot evaluate this model; they are its training distribution. Clean and degraded held-out renders both saturate above 0.98. Use real scans.
- Queries are handwritten semi-cursive and the corpus is printed regular script. Every number above describes that one pair of distributions.
- Fonts cover printed styles only. Real brush calligraphy differs structurally, not just in noise, and image degradation cannot synthesise it.
Failure worth recording
An ArcFace head over 23,440 classes was tried first and collapsed: render→corpus hit@1 fell 0.79 → 0.0035 on seen classes and 0.80 → 0.0050 on unseen after one epoch. Falling on both sides is collapse, not overfitting.
The cause is the data shape, not a hyperparameter: 23,440 classes with ~6 images each means a randomly initialised head never organises, so it back-propagates noise — and Adam normalises by gradient magnitude, so "lr = 1e-5" still destroys the pretrained structure within ~3,000 steps. Gradient clipping does not help; clipped noise is still noise.
Anyone retrying metric learning at this class-to-image ratio should expect the same.