Florence-2-base fine-tuned for Khmer Image Captioning
This model is a fine-tuned version of microsoft/Florence-2-base on the phonsobon/khmer_images_captioning_v2 dataset. It generates image captions in the Khmer language.
Model description
Florence-2 is a unified vision-language model from Microsoft that handles a range of vision tasks (captioning,
detection, segmentation, OCR, etc.) using task-specific prompts. This model was fine-tuned specifically on the
<MORE_DETAILED_CAPTION> task so that, given an image, it produces a detailed descriptive caption written in Khmer.
- Base model: microsoft/Florence-2-base
- Fine-tuning method: LoRA (via
peft) - Task prompt used:
<MORE_DETAILED_CAPTION> - Language: Khmer (km)
Intended uses & limitations
This model is intended for generating Khmer-language descriptions of general-scene photographs (landscapes, people, temples, markets, food, etc.), similar to the training distribution described below. It is not intended for:
- Safety-critical decision making
- Captioning of specialized domains not represented in the training data (e.g. medical imagery, satellite imagery)
- Tasks requiring perfect factual accuracy (like any captioning model, it can hallucinate details)
Since Florence-2's original tokenizer was not heavily pretrained on Khmer script, caption quality and fluency may vary; feedback and further fine-tuning on more data is welcome.
Training data
The model was fine-tuned on phonsobon/khmer_images_captioning_v2:
| Split | Rows |
|---|---|
| train | 3,690 |
| val | 977 |
| test | 978 |
Each example consists of an image paired with a Khmer-language caption (caption length: 57–384 characters).
Training procedure
- Base model: microsoft/Florence-2-base
- Fine-tuning method: LoRA (rank 8, alpha 16, dropout 0.05) applied to attention projection layers
- Task prompt:
<MORE_DETAILED_CAPTION> - Epochs: 3
- Effective batch size: 16 (batch size 4 × gradient accumulation 4)
- Learning rate: 2e-4 (LoRA)
- LR schedule: linear warmup (3%) + linear decay
- Max target length: 1024 tokens
- Precision: bf16 autocast
- Hardware: Kaggle GPU (T4 / P100)
How to use
import torch
from transformers import AutoProcessor, AutoModelForCausalLM
from peft import PeftModel
from PIL import Image
base_model_id = "microsoft/Florence-2-base"
adapter_id = "phonsobon/Images_captioning_fine_tune_Florence-2-base"
processor = AutoProcessor.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True, torch_dtype=torch.float32)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()
image = Image.open("your_image.jpg").convert("RGB")
task_prompt = "<MORE_DETAILED_CAPTION>"
inputs = processor(text=task_prompt, images=image, return_tensors="pt")
generated_ids = model.generate(
input_ids=inputs["input_ids"],
pixel_values=inputs["pixel_values"],
max_new_tokens=1024,
num_beams=3,
do_sample=False,
)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
caption = processor.post_process_generation(
generated_text, task=task_prompt, image_size=(image.width, image.height)
)[task_prompt]
print(caption)
Evaluation
Citation
If you use this model, please cite the base Florence-2 model and this dataset:
@article{xiao2023florence2,
title={Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks},
author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
journal={arXiv preprint arXiv:2311.06242},
year={2023}
}