ada-coder-qwen2.5-7b
Qwen/Qwen2.5-Coder-7B-Instruct fine-tuned with LoRA to write Ada 2022 and SPARK code from a task description, spec or signature.
[!WARNING] Review every line. The output is not qualified for DO-178C, EN 50128, ISO 26262 or any other certification process. Compile it, test it and, for SPARK, prove it before use.
🚀 Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "jgalego/ada-coder-qwen2.5-7b"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto")
messages = [
{"role": "system", "content": "You write Ada 2022 and SPARK code that compiles cleanly with GNAT."},
{"role": "user", "content": "Write a function that returns the greatest common divisor of two Positive numbers."},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))
It also completes raw Ada source with no chat template, which is how it is scored below.
🏋️ Training
| Base | Qwen/Qwen2.5-Coder-7B-Instruct |
| Data | r7_dataset.jsonl from the-clanker-lover/steelman-sft-ada, 6,110 instruction/code pairs that compile with GNAT |
| Method | LoRA, r=16, alpha=32, all linear layers, merged |
| Steps | 378 (1.0 epochs), batch size 16, max length 2048 |
| Learning rate | 0.0002, cosine |
| Mean train loss | 0.2128 |
| Hardware | NVIDIA A10G, 69 min |
📊 Results
nuprl/MultiPL-E HumanEval-Ada (humaneval-adb, 157 problems). Greedy decoding from the raw prompt, then built and run with the GNAT flags from MultiPL-E's eval_adb.py. A program passes when every pragma Assert holds.
| Model | n | pass@1 | Compile rate | Runtime errors | Timeouts |
|---|---|---|---|---|---|
| jgalego/ada-coder-qwen2.5-7b | 157 | 0.414 | 0.771 | 56 | 0 |
| jgalego/ada-coder-qwen2.5-7b@grpo-1 | 157 | 0.414 | 0.796 | 60 | 0 |
| jgalego/ada-coder-qwen2.5-7b@sft-2 | 157 | 0.401 | 0.771 | 58 | 0 |
| Qwen/Qwen2.5-Coder-7B-Instruct | 157 | 0.312 | 0.471 | 25 | 0 |
Reference scores reported by the Steelman authors (model card, report) on the same benchmark with their own harness, temperature 0:
| Model | pass@1 | Compile rate |
|---|---|---|
| Steelman 14B v0.3 (Qwen2.5-Coder-14B + R7 data) | 0.459 | 0.854 |
| Steelman 14B v0.2 (Qwen2.5-Coder-14B-Instruct + R6 data) | 0.471 | 0.752 |
| Qwen2.5-Coder-14B, untuned | 0.344 | 0.510 |
⚠️ Limitations
- Scored on small HumanEval-style functions, not on full programs or packages.
- Contracts are at most checked at run time (
-gnata); nothing here runs GNATprove. - Trained on a single dataset; style follows that data.