A 70M text-only encoder that maps a query into the embedding space of
Qwen3-VL-Embedding-8B,
distilled by pointwise cosine alignment against that teacher's cached query
embeddings. The -ML suffix marks the multilingual training mixture: 711K
English queries plus 778K MarianMT translations into five Latin-script European
languages.
Any half of the teacher, swapped
Both towers land in the same frozen teacher's embedding space, so either is a drop-in replacement for the corresponding half of that teacher. All four combinations retrieve; they differ in what you no longer have to run:
| teacher documents | student documents | |
|---|---|---|
| teacher queries | 71.05 — the ceiling, 8B on both sides | 65.02 — indexing 7x cheaper, teacher still runs per query |
| student queries | 66.36 — queries encode on one CPU thread, index built once by the teacher | 61.74 — no teacher anywhere |
Average NDCG@5 over ViDoRe v1+v2+v3, this model paired with NanoVDR-D-HiRes-Qwen3VL8B-4096, against Qwen3-VL-Embedding-8B.
The document towers in this space:
| NanoVDR-D-HiRes-Qwen3VL8B-4096 | 457M, 6 tiles, best quality |
| NanoVDR-D-Fast-Qwen3VL8B-4096 | 457M, 2 tiles, 2.7x faster indexing |
Usage
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nanovdr/NanoVDR-Q-DistilBERT-Qwen3VL8B-4096-ML")
q = model.encode(["What was the revenue growth in Q3 2024?"]) # (1, 4096), L2-normalised
Retrieval is a dot product against page vectors from a matching document tower:
from transformers import AutoModel, AutoImageProcessor
doc = AutoModel.from_pretrained("nanovdr/NanoVDR-D-HiRes-Qwen3VL8B-4096", trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained("nanovdr/NanoVDR-D-HiRes-Qwen3VL8B-4096", trust_remote_code=True)
d = doc.encode(pages, proc, batch_size=4)
scores = q @ d.T
Do not add an instruction prefix
Pass the raw query. The teacher's targets were cached with an instruction, but the student was trained to reproduce those targets from the bare query text, and every number below was measured that way. Prepending an instruction at inference moves the input off the distribution the tower was fitted on.
Results
NDCG@5 on ViDoRe. Query side in isolation, scored against teacher-encoded pages, which isolates this tower from document-side error:
| v1 | v2 | v3 | Avg | |
|---|---|---|---|---|
| NanoVDR-Q-DistilBERT-Qwen3VL8B-4096-ML | 84.68 | 64.30 | 50.09 | 66.36 |
| Qwen3-VL-Embedding-8B (teacher) | 87.31 | 69.76 | 56.07 | 71.05 |
End to end with the document towers, no teacher anywhere:
| Pairing | v1 | v2 | v3 | Avg |
|---|---|---|---|---|
| + NanoVDR-D-HiRes-Qwen3VL8B-4096 | 82.81 | 55.34 | 47.07 | 61.74 |
| + NanoVDR-D-Fast-Qwen3VL8B-4096 | 81.34 | 54.95 | 43.66 | 59.98 |
Query encoding costs 2.07 ms per query at batch 1 on an H200, including tokenisation. There is no vision model in the query path.
Backbone ablation
Measured under query-side isolation, all distilled from the same teacher:
| Backbone | Params | v1 | v2 | v3 | Avg | Latency |
|---|---|---|---|---|---|---|
| DistilBERT-base (this model) | 70M | 84.68 | 64.30 | 50.09 | 66.36 | 2.07 ms |
| BERT-base | 113M | 81.81 | 58.52 | 44.55 | 61.63 | 4.78 ms |
| ModernBERT-base | 152M | 85.36 | 65.40 | 51.43 | 67.40 | 10.61 ms |
ModernBERT is 1.04 points better for 2.2x the parameters and 5.1x the latency, which is why the small one ships as the default.
Training
| Objective | 1 - cos(student, teacher), nothing else |
| Teacher | Qwen3-VL-Embedding-8B, frozen, 4096-d |
| Data | 1.49M queries (the NanoVDR training set plus MarianMT translations) |
| Optimizer | AdamW, one-cycle, peak LR 5e-4, 3% warmup |
| Effective batch | 512 |
| Epochs | 15 |
Peak LR is half what the document tower uses; at 1e-3 a freshly initialised 768→4096 head diverges at the peak of the one-cycle schedule.
Reproducibility
Re-encoding the 500 ViDoRe arxivqa queries with this package and scoring
against the cached teacher page embeddings:
NDCG@5 teacher queries x teacher pages : 86.91
NDCG@5 student queries x teacher pages : 83.77 (96.4% retention)
which matches our internal evaluation for that dataset exactly.
Limitations
- English and five Latin-script European languages. The multilingual portion comes from a MarianMT translation pipeline. Non-Latin scripts such as Chinese, Japanese and Arabic are untested.
- Text-only. Image-conditioned queries are out of scope.
- Bounded by the teacher. Nothing in the objective lets the student exceed Qwen3-VL-Embedding-8B on the side it replaces.
Citation
@article{distilvdr2026,
title = {DistilVDR: A Compact End-to-End Visual Document Retriever
via Dual-Student Distillation},
author = {Liu, Zhuchenyang and Wang, Ziyi and Zhang, Yao and Xiao, Yu},
journal = {arXiv preprint arXiv:2608.10636},
year = {2026}
}
License
MIT.