Qwen3-VL-Embedding-2B-ko-vdr-distill
A reranker-distilled Korean Visual Document Retrieval (VDR) bi-encoder.
This model takes whybe-choi/Qwen3-VL-Embedding-2B-ko-vdr-preview-v0.2
(a Qwen3-VL-Embedding-2B contrastively fine-tuned on Ko-VDR) and further trains it by
listwise KL knowledge distillation from the Qwen3-VL-Reranker-8B cross-encoder
teacher — transferring the reranker's graded relevance (instead of binary labels) into
the 2B bi-encoder, which sharpens top-rank ordering.
Method (sequential CL → KD)
Following "Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation."
- Teacher:
Qwen/Qwen3-VL-Reranker-8B— scored{gold ∪ 128 mined candidates}(log-odds) for the 146,687 queries ofNomaDamas/ko-vdr-train-public-v2.0. - Objective: listwise KL,
KL( softmax(teacher/τ) ‖ softmax(student/τ) ), τ = 1.0.- Teacher scores are per-query max-subtract normalized (keeps the reranker's true score spread; min-max flattens it).
- Student score =
softmax(20 · cosine).
- Negatives: 8 per query, stratified via quantile anchors over the teacher scores (spans easy→hard, not just the hardest).
- Training: fresh LoRA (r = 32, α = 32) on a 50k-query subset, 1 epoch, lr 2e-5, bf16. No InfoNCE / no Matryoshka.
Evaluation (KoViDoRe-v2, cybersecurity)
At the recommended inference config (~1800 visual tokens + the prompts below):
| metric | baseline (v0.2) | distilled (this model) |
|---|---|---|
| NDCG@10 | 0.7629 | 0.7718 |
| MRR@10 | 0.8319 | 0.8775 |
| Recall@100 | 0.9809 | 0.9778 |
Distillation improves ranking quality — the MRR@10 gain (+0.046) is consistent across resolution/prompt settings — while Recall@100 is essentially unchanged. (Single-domain evaluation so far.)
Recommended inference config
processor_kwargs = {"max_pixels": 1843200, "min_pixels": 4 * 28 * 28} # ~1800 vision tokens
QUERY_PROMPT = "Find a document image that matches the given query."
DOCUMENT_PROMPT = "Represent the user's input."
Usage
import torch
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
"johnandru/Qwen3-VL-Embedding-2B-ko-vdr-distill",
model_kwargs={"attn_implementation": "flash_attention_2", "torch_dtype": torch.bfloat16},
processor_kwargs={"max_pixels": 1843200, "min_pixels": 4 * 28 * 28},
)
if hasattr(model[0], "unpad_inputs"):
model[0].unpad_inputs = False
q_emb = model.encode(["설계기준 강화 관련 문서"], prompt="Find a document image that matches the given query.")
d_emb = model.encode(["path/to/page.png"], prompt="Represent the user's input.") # image path
sim = model.similarity(q_emb, d_emb)
- Embedding dimension: 2048 · cosine similarity · bf16.
- Modalities: text query ↔ document page image.
Notes & limitations
- Preview / experimental. The distillation was trained at ~980 vision tokens with a generic prompt; a re-train at the recommended config (1800 tokens + the prompts above) would likely widen the gain further. Evaluation so far is single-domain.
- Data: Korean public-sector documents (49 PDFs, synthetic queries via Solar Pro 3).