Vesper-Coder-1.5B (v1, Gen-2 Checkpoint)

Vesper-Coder-1.5B (v1) is an experimental 1.5B-parameter dense code generation checkpoint initialized from Qwen/Qwen2.5-Coder-1.5B-Instruct and trained via two rounds of on-policy Direct Preference Optimization (DPO) on a 464-task MBPP training pool.

Important Evaluation Update (Normalized Harness Audit): Under an EvalPlus-normalized evaluation harness (max_new_tokens=384 + AST post-processing sanitizer) with strictly leak-free prompt-example gating, the Vesper-Coder-1.5B (v1, Gen-2) weights do not improve over Base Qwen2.5-Coder-1.5B-Instruct (all paired two-sided McNemar $p > 0.05$; slightly trailing base on raw counts).

Earlier gains reported under our initial harness (max_new_tokens=256, no AST sanitizer: 54.86% vs. 47.47% on MBPP; 63.41% vs. 58.54% on HumanEval) were an artifact of the 256-token generation cap truncating verbose Base Qwen2.5-Coder-1.5B-Instruct outputs. Raising max_new_tokens to 384 and applying standard AST sanitization increased Base Qwen2.5-Coder-1.5B-Instruct 1-shot accuracy by +10.51 pp on MBPP (47.47% -> 57.98%) and +6.70 pp on HumanEval (58.54% -> 65.24%), eliminating the v1 weight advantage. We retain this checkpoint publicly as a reproducible artifact and negative-result audit for small-model on-policy DPO.

1. Checkpoint Provenance & How DPO Preference Pairs Were Selected

• Exported Checkpoint: Gen-2 (second generation of iterative on-policy LoRA DPO, $r=16, \alpha=32$, merged into dense bfloat16 weights via merge_and_unload()). In the initial v1 experiments, Gen-1 (55 steps) and Gen-2 (115 steps) were evaluated directly on the test sets without a separate validation split; the uploaded weights in this repository are Gen-2.
• Training Pool (N = 464 Tasks): Constructed from the full google-research-datasets/mbpp dataset (974 tasks) by removing all 257 task IDs belonging to the official MBPP-Sanitized test split (Train ∩ MBPP-Sanitized Test = ∅). Note that 147 of the 378 tasks in EvalPlus MBPP+ overlap with this 464-task pool; therefore, MBPP+ is reported both on all 378 tasks and on the strictly disjoint N = 231 subset.
• How DPO Winners (y_win) and Losers (y_lose) Were Selected:
  1. For each task in the 464-task training pool, the model first generated a 1-shot greedy (T = 0.0, max_new_tokens = 256) solution.
  2. Solutions were executed in a sandbox against the training task's ground-truth test_list assertions.
  3. Whenever the T = 0.0 attempt failed (y_lose), up to 2 retry turns were sampled (T = 0.40–0.60, max_new_tokens = 256) conditioned on the sandbox execution error and a truncated function-header prefix. Any retry that passed all test_list assertions was recorded as a verified winner (y_win), forming an on-policy preference pair (prompt, y_win > y_lose).
  4. Why v1 DPO Did Not Transfer Under the 384-Token Normalized Harness: Because rollouts during v1 harvesting were capped at max_new_tokens = 256, many y_lose trajectories failed simply due to token truncation at 256 tokens, and y_win selected shorter responses that fit inside 256 tokens. Once evaluation was normalized to max_new_tokens = 384 with AST sanitization, Base Qwen2.5-Coder-1.5B-Instruct no longer suffered from 256-token truncation.

2. Normalized-Harness Benchmark Results (max_new_tokens=384 + AST Sanitizer)

All evaluations below use identical generation caps (max_new_tokens=384), identical AST sanitization, and strictly leak-free test-time retry gating (retry loops execute only the public example already shown in the prompt: test_list[0] on MBPP and docstring >>> examples on HumanEval).

A. 1-Shot (T = 0.0) & 3-Turn Retry Accuracy Across 6 Suites

B. Paired Two-Sided Exact McNemar Tests: Vesper-Coder-1.5B (Gen-2) vs. Base Qwen2.5-Coder-1.5B-Instruct

Here b counts tasks where Vesper passes and Base fails; c counts tasks where Base passes and Vesper fails:

3. Usage (transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "axieyangb/Vesper-Coder-1.5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Write a Python function `longest_increasing_subsequence(nums: list[int]) -> int` that returns the length of the longest strictly increasing subsequence."
messages = [
    {"role": "system", "content": "You are an expert Python programmer. Output clean, well-structured Python code."},
    {"role": "user", "content": prompt},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=384, do_sample=False)

response = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

License

Released under the Apache-2.0 License.