ko-vdr-preview

Korean visual document retrieval — 6 MTEB multimodal tasks (text→image).

A LoRA fine-tune of Qwen/Qwen3-VL-Embedding-2B trained on a mixed Korean/English VDR corpus with hard negatives mined by Qwen3-VL-Embedding-8B. Supports Matryoshka embeddings down to 128 dimensions (default: 2048).

Summary of Findings

• Significant Improvement over 2B: ko-vdr-preview shows a massive performance uplift compared to the Qwen3-VL-2B baseline (e.g., ~0.48 vs ~0.35 avg nDCG@10).
• Closing the Gap with 8B: The model's performance is remarkably close to the Qwen3-VL-8B model, offering near-state-of-the-art accuracy with much greater efficiency.

Usage

Install Dependencies

pip install -U sentence-transformers>=5.4.1
### Python code
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("johnandru/ko-vdr-preview")

# Run inference
queries = [
    '30인 이상 상용근로자를 보유한 기업의 1인당 평균 월별 법정외 복지비용이 10~29인 규모 기업보다 높은지 판단해 주세요'
]
documents = [
    'ko-vdr-public/3818.png',
    'ko-vdr-public/7753.png',
    'ko-vdr-public/3760.png'
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 2048] [3, 2048]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)

Matryoshka truncation

The model supports shortened embeddings via Matryoshka training. Supported dimensions: 2048, 1536, 1024, 768, 512, 256, 128 model = SentenceTransformer("johnandru/ko-vdr-preview", truncate_dim=512)

Model Details

Training

Data

Training used a multi-source Korean/English VDR dataset with hard negatives mined offline:

Hard negatives were mined with Qwen/Qwen3-VL-Embedding-8B using absolute_margin=0.05 and 7 negatives per pair (top sampling).

Loss

MatryoshkaLoss(SelfGuideCachedMultipleNegativesRankingLoss): InfoNCE with cosine similarity (scale=20), cached mini-batches (mini_batch_size=4), and Matryoshka multi-granularity weighting.

Evaluation

Task abbreviations

Results - nDCG@10

Results - Recall@10

Notes

• All Qwen3-VL-Embedding family models loaded with max_pixels = 1280 * 28 * 28, bf16, flash-attention-2.
• Prompt usage:
  • Qwen3-VL-Embedding 2B / 8B and our LoRA fine-tune: training prompt "Represent the user's input." (matches train.py).
• LoRA fine-tune used peft 0.19.1 workaround in loader.py to inject lora_B weights (transformers 5.5.4 silently dropped them on from_pretrained for headless models — see PR huggingface/transformers#45428).