Objective
Fine-tune PaddlePaddle/PaddleOCR-VL on WildKhmerST.
For future benchmarking in Khmer OCR task, WildKhmerST is processed to contains pairs of 'text region image' and 'text'.
The processed dataset is uploaded to huggingface platform here.
There are 3 splits. train split is used to full fine-tune PaddlePaddle/PaddleOCR-VL.
test split is used to evaluate the fine-tuned model.
Uploaded model
- Developed by: Kimang18
- License: apache-2.0
- Finetuned from model : paddle_ocr_ft
This paddleocr_vl model was trained 2x faster with Unsloth
Usage
Inference using Huggingface transformers on NVIDIA GPUs.
Requirements tested on python 3.12.13, T4 GPU:
transformers<5.0.0
from PIL import Image
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
# ---- Settings ----
model_path = "Kimang18/paddleOCR_vl_Khmer_finetuned"
image_path = "khmer_text_region.png"
# ------------------
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
image = Image.open(image_path).convert("RGB")
model = AutoModelForCausalLM.from_pretrained(
model_path, trust_remote_code=True, dtype=torch.bfloat16
).to(DEVICE).eval()
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
messages = [
{"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": "OCR:"},
]
}
]
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
).to(DEVICE)
outputs = model.generate(**inputs, max_new_tokens=1024)
outputs = processor.batch_decode(outputs, skip_special_tokens=True)[0]
print(outputs)
NOTE: if you use transformers>=5.0.0, you will encounter KeyError from rope_init_fn=ROPE_INIT_FUNCTIONS[rope_type].
This is because PaddleOCR-VL uses rope_type="default" which is not supported anymore for transformers>=5.0.0.