qwen2.5-coder-7b-apps-qlora
A QLoRA fine-tune of Qwen2.5-Coder-7B-Instruct on the APPS competitive
programming training split, produced as a portfolio study of before/after
performance on execution-based code benchmarks. This is the merged 16-bit model;
the LoRA adapter is available separately at
Shaurya-saini/qwen2.5-coder-7b-apps-qlora-lora.
Important, read first. In a controlled, difficulty-stratified evaluation on the held-out APPS test split, this fine-tune underperforms its own base model, Qwen2.5-Coder-7B-Instruct, on every difficulty tier. It is published for transparency and reproducibility, not as an improvement over the base. If you need the stronger model, use the base. See Evaluation and Limitations below.
Model details
- Base model: Qwen/Qwen2.5-Coder-7B-Instruct
- Method: QLoRA (4-bit base, LoRA rank 16 on attention and MLP projections), trained with Unsloth
- Training data: APPS training split (one solution per problem, formatted as QUESTION/ANSWER inside the Qwen chat template)
- Training: 1 epoch, effective batch size 8, ~620 steps, single Kaggle T4, final training loss ~0.65
- License: Apache-2.0 (inherited from the base model)
Intended use and prompt format
Intended for Python competitive-programming style problems. The model was trained inside the Qwen chat template, so it must be prompted with the chat template (using a bare completion prompt drives it off-distribution and degrades output).
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "Shaurya-saini/qwen2.5-coder-7b-apps-qlora"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, device_map="auto")
prompt = "QUESTION:\n<your problem statement>\n\nUse Standard Input format\nANSWER:\n"
msgs = [{"role": "user", "content": prompt}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(input_ids=inputs, max_new_tokens=1024, eos_token_id=tok.convert_tokens_to_ids("<|im_end|>"))
print(tok.decode(out[0], skip_special_tokens=True))
Evaluation
APPS held-out test split, 150 problems per difficulty tier, evaluated with bigcode-evaluation-harness. Both models were run identically: 4-bit weights, the same Qwen chat-template prompt, generated code executed against hidden tests. pass@1 is strict accuracy (all hidden tests must pass); the value in parentheses is the average fraction of individual test cases passed.
| Difficulty | Base (Qwen2.5-Coder-7B-Instruct) | This model (fine-tuned) |
|---|---|---|
| Easy / Introductory | 0.0% (0.84%) | 0.0% (0.22%) |
| Medium / Interview | 7.3% (30.2%) | 2.0% (7.4%) |
| Hard / Competition | 0.0% (0.54%) | 0.0% (0.0%) |
Limitations
The fine-tune regressed relative to the base model. The main causes, confirmed by inspecting generations:
- Training data selection. Training used the single shortest solution per problem; the shortest APPS solutions are golfed, cryptic code. Imitating them narrowed the model toward terse, cruder attempts.
- Catastrophic forgetting. One epoch on a narrow, stylistically-skewed target eroded the base model's broader coding ability.
- A learned syntax artifact. The model over-produces closing brackets on some problems, causing compile errors the base model avoids.
A follow-up (v2) will address these by selecting cleaner/longer solutions, lowering the learning rate, and re-evaluating on the same split.
Reproducibility
Full training and evaluation code, the complete issue-and-fix log, and the analysis are in the project repository. Training, upload, and evaluation were run in public Kaggle notebooks (linked from the project README).