SmolVLM2-500M-vi-stage1
Vietnamese stage-1 adaptation of
SmolVLM2-500M-Video-Instruct.
The tokenizer was extended with 8,064 Vietnamese sub-word tokens
(vocab 49,280 → 57,344) and the model was trained on a Vietnamese
image–text mixture. This is a standalone merged model — the LoRA adapter
and the newly-trained token embeddings have been folded into the base weights,
so it loads directly with AutoModelForImageTextToText.from_pretrained with no
adapter step.
What was trained
- Tokenizer expansion: 8,064 new Vietnamese BPE tokens (IDs 49,280–57,343), mean-initialized, cutting Vietnamese fertility (tokens/word) substantially.
- Recipe: LoRA (r=32, α=64) on attention + MLP projections, plus only the
8,064 new token rows of
embed_tokensandlm_headtrained via PEFTtrainable_token_indices(base vocabulary frozen). This avoids the base-vocab forgetting / mode collapse that fully training the embeddings caused in an earlier run. - Schedule: 2 epochs, effective batch 48 (4 × 3 GPU × 4 grad-accum), LR 1e-4 cosine, bf16, 1536px images.
Training data
Vietnamese multimodal + a slice of English, roughly:
| Source | Type | Note |
|---|---|---|
| ViOCRVQA | OCR-VQA (vi) | ~19.7k |
| OpenViVQA | scene-text VQA (vi) | ~9.1k |
| UIT-ViIC | image captioning (vi) | ~13.5k |
| the_cauldron (vqav2 / ocrvqa / textvqa) | VQA (en) | sampled, anti-forgetting |
| Vietnamese Wikipedia | text | ~25k, signal for new embeddings |
Evaluation
Base vs. this model, 200 samples/task (exact_match/token_f1 for VQA;
bleu/rougeL for captioning):
| Task | Metric | Base | This model |
|---|---|---|---|
| ViOCRVQA (test) | token_f1 | 5.31 | 7.04 |
| ViOCRVQA (test) | exact_match | 0.00 | 1.50 |
| UIT-ViIC (valid) | rougeL | 12.29 | 26.83 |
| UIT-ViIC (test) | rougeL | 13.39 | 21.94 |
| OpenViVQA (dev) | token_f1 | 19.73 | 3.17 |
The main, genuine gains are in Vietnamese image captioning (UIT-ViIC rougeL
roughly doubles) and modest OCR-VQA improvement. The OpenViVQA token_f1
number drops, but note the base score there is inflated by the base model
echoing the question back (which overlaps reference tokens); both models
actually struggle with this open-ended scene-text VQA task at 500M scale.
Usage
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
repo = "thuanan/SmolVLM2-500M-vi-stage1"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(
repo, torch_dtype=torch.bfloat16
).to("cuda").eval()
image = Image.open("your_image.jpg").convert("RGB")
messages = [{"role": "user", "content": [
{"type": "image"},
{"type": "text", "text": "Mô tả bức ảnh này."},
]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to("cuda", torch.bfloat16)
out = model.generate(**inputs, max_new_tokens=64)
print(processor.batch_decode(out, skip_special_tokens=True)[0])
Limitations
- Stage-1 only: best at captioning; open-ended Vietnamese VQA (esp. scene-text / OCR reasoning) remains weak — expected for a 500M model.
- Occasional spelling artifacts on rare Vietnamese tokens (under-trained embeddings), most visible on OpenViVQA-style questions.
- English capability is partially preserved (anti-forgetting data) but not benchmarked here.
License
Apache-2.0, inherited from the base SmolVLM2 model.