AutoDataBench Retrieval Resources
Resources for the retrieval task in AutoDataBench. See the paper for the benchmark setting.
Contents
data/retrieval_v1/train.jsonl
models/MiniLM-L6-H384-uncased/
models/Qwen3-4B-Instruct-2507/
models/Qwen3-Embedding-0.6B/
train.jsonl is the 818,182-row source pool available to the data agent. Each
row contains query, positive_doc, hard_negative_docs, and subset.
| Model | Role | Original model |
|---|---|---|
| MiniLM-L6-H384-uncased | Fixed retrieval base model | nreimers/MiniLM-L6-H384-uncased |
| Qwen3-4B-Instruct-2507 | Agent-callable generation model | Qwen/Qwen3-4B-Instruct-2507 |
| Qwen3-Embedding-0.6B | Agent-callable embedding model | Qwen/Qwen3-Embedding-0.6B |
The fixed and OOD evaluations use standard MTEB datasets, which are downloaded by MTEB and are not duplicated here. The task configuration names the exact evaluation suites.
Download
Download the complete resource bundle with huggingface_hub:
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="AutoDataBench/Retrieval-resources",
local_dir="./Retrieval-resources",
)
Use with AutoDataBench
Copy or symlink data/ and models/ into the AutoDataBench repository. The
default config refers to the auxiliary models by Hugging Face ID; point the
model servers at the local directories above for a fully local setup.
MANIFEST.sha256 contains checksums for every distributed file.
Model and dataset components retain their upstream licenses. Consult the model cards and source datasets before redistribution or commercial use.
Citation
If you use these resources, please cite:
@misc{yuan2026autodatabench,
title = {AutoDataBench: A Data-centric Testbed for Accelerating Auto Research},
author = {Ruifeng Yuan and Yizhi Li and Yaxin Du and Fengyu Cai and Yiqi Liu and Hou Pong Chan and Chenghua Lin and Yun Chen and Jian Yang and Bryan Dai and Pinyan Lu and Chenghao Xiao},
year = {2026},
eprint = {2609.40097},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2609.40097}
}