Objective
Fine-tune deepseek-ai/DeepSeek-OCR on WildKhmerST.
The setting for image processing is the following
base_size=512
image_size=512
crop_mode=False
The setting for LoRA is the following
model = FastVisionModel.get_peft_model(
model,
target_modules = [
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
r = 16, # The larger, the higher the accuracy, but might overfit
lora_alpha = 16, # Recommended alpha == r at least
lora_dropout = 0,
bias = "none",
random_state = 3407,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
# target_modules = "all-linear", # Optional now! Can specify a list if needed
)
For future benchmarking in Khmer OCR task, WildKhmerST is processed to contains pairs of 'text region image' and 'text'.
The processed dataset is uploaded to huggingface platform here.
There are 3 splits. train split is used to fine-tune unsloth/DeepSeek-OCR.
test split is used to evaluate the fine-tuned model.
The fine-tuned model performs better on Khmer text, yet it still completely misses some images in the test split.
Concretely, its cer on the whole test split is 237% (result file can be found here)
This is because when the fine-tuned completely misses decoding some images, it hallucinates and then repeats the same text token until the number of maximum tokens is met.
Removing hallucinating cases, cer drops to 67.66% (result file can be found here).
Uploaded finetuned model
- Developed by: Kimang18
- License: apache-2.0
- Finetuned from model : deepseek-ai/DeepSeek-OCR
This deepseek_vl_v2 model was trained 2x faster with Unsloth and Huggingface's TRL library.
Usage
TBA