Elixir Qwen2.5 Coder 3B
A small Elixir coding model built from Qwen2.5-Coder-3B-Instruct. One transformer block was added (36 layers to 37, about 3.16B parameters), then the model was fine-tuned for Elixir. It is meant to run locally: the Q8 GGUF is a little over 3 GB.
Modified derivative of Qwen2.5-Coder-3B-Instruct. The architecture and weights in this repository are not the original Qwen release. Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved. See
LICENSEandNOTICE.
Elixir coding results
Scored on a 128-task executable harness. Each task asks for one Elixir module. A pass means the completion parses, compiles, and its tests pass on Elixir 1.20.2 / OTP 29. One completion per task, temperature 0, 16,384-token context, llama.cpp. Both models below are Q8_0 GGUF, scored in the same run.
| Qwen2.5-Coder-3B-Instruct | Elixir Qwen2.5 Coder 3B | Change | |
|---|---|---|---|
| Tasks passed | 31 / 128 | 45 / 128 | +14 |
| Pass rate | 24.2% | 35.2% | +10.9 pp |
| Relative gain | +45% |
Where the gain shows up
| Area | Base | Elixir Qwen2.5 Coder 3B |
|---|---|---|
| Binaries | 0 / 16 | 5 / 16 |
| Abstractions | 5 / 16 | 9 / 16 |
| Core language | 3 / 16 | 6 / 16 |
| Collections | 4 / 16 | 6 / 16 |
Warning-free completions rise from 60.2% to 74.2%, and the output-format rate reaches 100%.
Run it
llama.cpp
llama-server -m Elixir-Qwen2.5-Coder-3B-Q8_0.gguf -c 16384 --temp 0.7 --top-k 20 --top-p 0.8 --repeat-penalty 1.05
The benchmark file is Elixir-Qwen2.5-Coder-3B-Q8_0.gguf.
Transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "jmarceno/Elixir-Qwen2.5-Coder-3B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
messages = [
{
"role": "user",
"content": "Write an Elixir module Counter with a GenServer that stores an integer and supports increment and get.",
}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
License
Research and evaluation use follows the Qwen Research License. Commercial use of the Qwen materials needs a separate license from Alibaba Cloud. This repository redistributes those materials with the extra layer and the Elixir fine-tune described above.