Temper-1-1.5B
Temper-1-1.5B is Qwen2.5-Coder-1.5B-Instruct fine-tuned for writing Rust. On the standard HumanEval-Rust benchmark it solves 50.5% of tasks on the first try, more than Qwen2.5-Coder-3B at twice its size.
The model was trained on the same 10 000 Rust examples as Temper-1-0.5B. Each example was written by DeepSeek-V4-Flash and passed the compiler and tests. This model had no reinforcement learning.
Website: temper-ai.pages.dev/temper-1.5b
Results
HumanEval-Rust (multiple-rs from MultiPL-E, 156 tasks), %. pass@1 is the share of tasks solved on the first try, pass@10 the share solved in at least one of 10 tries.
| Model | Size | pass@1 | pass@10 |
|---|---|---|---|
| Qwen2.5-Coder-0.5B | 0.5B | 16.0 | 27.9 |
| StarCoder2-3B | 3B | 25.1 | 37.8 |
| Temper-1-0.5B | 0.5B | 25.2 | 38.0 |
| Qwen2.5-Coder-1.5B | 1.5B | 39.3 | 56.6 |
| Qwen2.5-Coder-3B | 3B | 44.7 | 67.4 |
| Temper-1-1.5B | 1.5B | 50.5 | 67.0 |
| Qwen3.5-9B | 9B | 56.7 | 73.0 |
| Qwen2.5-Coder-7B | 7B | 57.3 | 74.3 |
Every model ran through bigcode-evaluation-harness at commit 8fc5bae with the BigCode Models Leaderboard settings. The model completes code without a chat template, with temperature 0.2, top_p 0.95, 50 samples per task and 512 tokens including the prompt. Qwen2.5-Coder and StarCoder2 are the base versions. Qwen3.5-9B is the post-trained version. StarCoder2-3B scores 25.1 here and 24.5 on the leaderboard.
The Temper-1-0.5B card reports HumanEval-RS with the task sent as a chat message, so its numbers there differ from this table.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "temper-ai/Temper-1-1.5B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Write a Rust function that returns the n-th Fibonacci number."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
The default generation config samples with temperature 0.7, top_p 0.8, top_k 20 and repetition penalty 1.1.
A GGUF version for llama.cpp, LM Studio and Ollama is in temper-ai/Temper-1-1.5B-GGUF.
Limitations
With ten tries Temper-1-1.5B solves 67.0% of tasks, about the same as Qwen2.5-Coder-3B with 67.4%. The model was trained only on Rust tasks with requests in English. Requests in other languages and questions unrelated to Rust may get made-up answers or refusals. It was not trained for fill-in-the-middle, so it is not a drop-in model for inline completion in an editor. The training data has few tasks on async code and web frameworks.
License
Apache 2.0, same as Qwen2.5-Coder-1.5B-Instruct.