daVinci-Dev-72B

Table of Contents

• News
• Overview
• Key Results
• Model Zoo
• Datasets
• Pipeline
• Quick Start
• Training
• Evaluation
• License
• Citation

News

• 2026-05: 🎉 daVinci-Dev was accepted as an oral presentation at ICML 2026.
• 2026-01: daVinci-Dev paper, code and dataset were released!

Overview

daVinci-Dev is a family of large language models trained for agentic software engineering.

This work presents a systematic study of agentic mid-training and introduces agent-native data to reduce the distribution mismatch between static pretraining corpora and the dynamic, feedback-rich environments faced by real code agents.

Our training uses two complementary trajectory types (details in the paper):

• Contextually-native trajectories \(\mathcal{D}^{\text{ctx}}_{\text{py}}\) (PR-derived): preserve the full information flow by bundling file discovery/context retrieval together with sequential edits. This provides broad coverage and diversity.
• Environmentally-native trajectories \(\mathcal{D}^{\text{env}}_{\text{pass}}\) (executable rollouts): collected from real executable repositories with genuine tool/test outputs, capturing authentic feedback loops.

Resources (open-source / open-release):

• Paper + data processing pipeline: https://github.com/GAIR-NLP/daVinci-Dev
• Dataset: https://huggingface.co/datasets/GAIR/daVinci-Dev

Key Results

SWE-Bench Verified

We reach SOTA among open training recipes using agentic scaffolds under their model sizes, despite starting from the non-coder Qwen2.5-Base family.

Generalization gains: improvements are also observed on standard code benchmarks (e.g., HumanEval/EvalPlus) and scientific reasoning benchmarks (e.g., GPQA/SciBench) as reported in the paper.

Model Zoo

We will open-source model checkpoints on Hugging Face:

Datasets

We will open-source our datasets through Hugging Face:

Pipeline

The GitHub repository contains a high-performance pipeline that calls the GitHub API and constructs the structured PR representation used to build \(\mathcal{D}^{\text{ctx}}_{\text{py}}\).

Quick Start

These checkpoints are intended to be used inside the SWE-Agent scaffold. They are also compatible with standard inference frameworks.

pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "GAIR/daVinci-Dev-72B"  # or any checkpoint in the model zoo

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": (
            "You are a software engineering agent. "
            "When solving tasks, reason about the repo structure, propose minimal edits, "
            "and describe how you would validate with tests."
        ),
    },
    {"role": "user", "content": "Bug: tests fail when X. Please fix it."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=2048,
        temperature=0.2,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model GAIR/daVinci-Dev-72B \
  --tensor-parallel-size 8 \
  --max-model-len 131072

Training

This section summarizes the methodology described in the paper.

Data

• Contextually-native PR trajectories: 68.6B tokens (constructed from GitHub pull requests, preserving the coupling between context retrieval and edits).
• Environmentally-native executable trajectories: 3.1B raw tokens (4.5B effective tokens), collected by running an agent in real executable environments with tool and test feedback. Trajectories include both test-passing and non-passing rollouts.

Recipe (high level)

• Start from the Qwen2.5 base model family (32B / 72B).
• Perform agent-native mid-training on PR-derived trajectories (and optionally mixed with executable trajectories).
• Perform SFT on the test-passing subset of environmentally-native trajectories.

-MT checkpoints correspond to the state after mid-training and before SFT.

Evaluation

We report performance on SWE-Bench Verified using SWE-Agent with the setup described in the paper (including temperature 0, 128k context, and a 100-step budget). Results are reported as Pass@1 (averaged across 4 runs).

License

This project is a mixed release:

• Contextually-native PR-derived subset: only PRs from repositories detected as having a permissive license are included. Each repo’s license is provided in ./ctx-native/filtered_repos/part-0000.parquet.
• Environmentally-native subset: derived from SWE-rebench, licensed under CC-BY-4.0.
• daVinci-Dev models: released under Qwen license. Users should verify the licensing status of any generated code before using it in production.
• daVinci-Dev pipeline: released under the Apache-2.0 license.

Users are responsible for ensuring their downstream usage complies with the licenses of the underlying sources.

Citation

If you use this work, please cite the daVinci-Dev paper.

@misc{zeng2026davincidevagentnativemidtrainingsoftware,
      title={daVinci-Dev: Agent-native Mid-training for Software Engineering},
      author={Ji Zeng and Dayuan Fu and Tiantian Mi and Yumin Zhuang and Yaxing Huang and Xuefeng Li and Lyumanshan Ye and Muhang Xie and Qishuo Hua and Zhen Huang and Mohan Jiang and Hanning Wang and Jifan Lin and Yang Xiao and Jie Sun and Yunze Wu and Pengfei Liu},
      year={2026},
      eprint={2601.18418},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2601.18418},
}