Koda-0.5B
A fine-tuned version of Qwen2.5-Coder-0.5B-Instruct for Python code
generation from natural-language instructions. This is the full merged
model — LoRA weights merged into the base, so it loads directly with
transformers, no PEFT needed.
Model details
- Base model: Qwen/Qwen2.5-Coder-0.5B-Instruct by the Qwen team (Alibaba Cloud) — https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct
Training details
| Setting | Value |
|---|---|
| Method | LoRA via PEFT (merged into base after training) |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable params | ~8.8M (1.75% of 0.5B) |
| Dataset | iamtarun/python_code_instructions_18k_alpaca (subset) |
| Train / eval examples | 1,400 / 100 |
| Epochs | 2 |
| Learning rate | 2e-4, cosine schedule, 20 warmup steps |
| Batch size | 1 × 8 grad-accum steps (effective 8) |
| Precision / device | bfloat16, CPU-only |
| Training time | ~2.26 hours (~48 tokens/sec) |
Results
- Final train loss: 0.78
- Final eval loss: 0.82 (stable across epochs — no overfitting observed)
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "pooraddyy/Koda-0.5B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
model.eval()
prompt = (
"<|im_start|>user\n"
"Write a python function to sort a list of dictionaries by age.\n"
"<|im_end|>\n<|im_start|>assistant\n"
)
inputs = tok(prompt, return_tensors="pt")
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=120, do_sample=False,
pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Expected output:
def sort_by_age(list_of_dicts):
# Sort the list of dictionaries by age
list_of_dicts.sort(key=lambda x: x['age'])
return list_of_dicts
Intended use
- Generating short Python functions/snippets from English instructions.
- Learning / experimentation with small-model fine-tuning.
Limitations
- 0.5B parameters is a small model — keep expectations modest. It can produce incorrect or insecure code.
- Trained only on short Python instruction examples; performance on other languages, long programs, or complex reasoning is limited.
- Always review generated code before running it.
License
Apache 2.0. This model is derived from Qwen2.5-Coder-0.5B-Instruct — all credit for the base model belongs to the Qwen team (Alibaba Cloud), and the base model's original license and attribution are retained.