ZY-S2 by ZYR AI
ZY-S2 is a 1.5B coding model from ZYR AI. It is a LoRA fine-tune of Qwen2.5-Coder-1.5B with a focus on writing correct Python. It runs on a laptop CPU with llama.cpp (Q5_K_M GGUF, about 1.1 GB).
Files: model.safetensors (merged full weights, Transformers), ZY-S2-Q5_K_M.gguf (llama.cpp).
Benchmarks
Rows in the first table: all three models were run as Q5_K_M GGUF on CPU with llama.cpp llama-server, using the same harness and prompts.
| Benchmark | ZY-S2 (Q5_K_M) | Qwen2.5-Coder-0.5B-Instruct (Q5_K_M) | Qwen2.5-Coder-1.5B-Instruct (Q5_K_M) |
|---|---|---|---|
| HumanEval pass@1 (164) | 75.6 | 54.3 | 68.9 |
| HumanEval+ pass@1 (164) | 68.9 | 50.0 | 65.2 |
| SLMFrontier coding (first 20 items) | 95.0 | 80.0 | 90.0 |
| SLMFrontier math (first 20 items) | 35.0 | 25.0 | 40.0 |
| SLMFrontier reasoning (first 20 items) | 20.0 | 25.0 | 5.0 |
| SLMFrontier instruction (first 20 items) | 42.1 | 37.5 | 48.8 |
| SLMFrontier tools (first 20 items) | 50.0 | 35.0 | 95.0 |
| Computer-use sim, 60 tasks, end-to-end success % | 13.3 | 6.7 | 20.0 |
| Security review recall % (10 planted flaws) | 70.0 | 80.0 | 90.0 |
| Insecure code rate % (20 prompts, lower=better) | 65.0 | 45.0 | 70.0 |
| Refuses malicious requests % | 86.7 | 66.7 | 93.3 |
| Helps with legit security requests % | 100.0 | 100.0 | 100.0 |
SLMFrontier, full 9-area run
SLMFrontier Score = mean of 9 areas, 95% bootstrap CI. Per-item results: benchmarks/ (benchmarks/slmfrontier-c67/, benchmarks/coding_compare/).
| Area (full set) | n | ZY-S2 (Q5_K_M, CPU llama.cpp) | Qwen2.5-Coder-1.5B (Q4_0, GPU 2x T4) |
|---|---|---|---|
| SLMFrontier Score | 37.87 (36.58–39.25) | 46.34 | |
| coding (HumanEval+ / MBPP+) | 400 | 68.0 | 61.5 |
| math (MATH-500, GSM8K, AIME24/25) | 460 | 27.61 | 38.0 |
| reasoning (BBH, 10 hard subtasks) | 400 | 24.25 | 22.5 |
| instruction (IFEval) | 400 | 36.38 | 40.8 |
| tools (BFCL v3) | 400 | 40.5 | 72.8 |
| long context (NIAH + var tracking, 4k–16k) | 300 | 81.0 | 96.3 |
| agentic (multi-turn python tool tasks) | 200 | 4.5 | 0.0 |
| knowledge (MMLU-Pro chance-adjusted, SimpleQA) | 500 | 8.2 | 6.8 |
| safety (XSTest balanced refusal) | 450 | 50.4 | 78.3 |
Plainly: Qwen2.5-Coder-1.5B has the higher overall score and is well ahead on tools, long context and safety. ZY-S2 wins coding. The two runs used different setups (ZY-S2: Q5_K_M on CPU; Qwen: Q4_0 on GPU), so this is not an exact comparison. Agentic is exact-match task success and likely understates both models. Knowledge is chance-adjusted, so scores near 0 mean near-chance. ZY-S2's safety is about 50 because it answers almost every prompt and rarely refuses unsafe ones.
SLMFrontier coding, same harness for every model (CPU llama.cpp, all Q5_K_M)
| Model | Coding (400) | HumanEval+ (163) | MBPP+ (237) | 95% CI |
|---|---|---|---|---|
| ZY-S2 | 68.0 | 74.2 | 63.7 | 63.5–72.8 |
| Qwen2.5-Coder-1.5B-Instruct | 63.5 | 63.2 | 63.7 | 58.8–68.2 |
| Qwen2.5-Coder-0.5B-Instruct | 47.0 | 52.1 | 43.5 | 42.2–52.0 |
| Llama-3.2-1B-Instruct | 38.0 | 27.0 | 45.6 | 33.2–42.8 |
The confidence ranges of ZY-S2 and Qwen2.5-Coder-1.5B overlap slightly.
Methodology
- HumanEval / HumanEval+: EvalPlus problems and official EvalPlus test runner. Each problem goes through the chat template with the EvalPlus instruction ("Please provide a self-contained Python script...") and a fixed code-block response prefix. Decoding is greedy (temperature 0, top-k 1), with up to 768 new tokens and pass@1. ZY-S2 gets an empty
<think></think>block before the prefix because it was trained with that format. The Qwen models get the same prompt without that block. - SLMFrontier is our benchmark covering HumanEval+/MBPP+, MATH-500/GSM8K/AIME, BBH, IFEval and BFCL v3. Decoding is greedy, and every model gets the same system prompt. The three-way rows use only the first 20 items of each area (the same items for every model), so each row is a small sample and noisy (±10 points or more). Full-set scores are in the tables below it; prefer those over the 20-item rows.
- Computer-use sim: our 60-task simulated shell + web-form agent eval (up to 8 steps per task, checked by a task checker).
- Security tests: a sandboxed, intentionally vulnerable demo Flask app with fake data and no network. The tests are vulnerability review (recall of 10 planted flaws), secure-code generation (20 prompts, judged by rules), and refusal of malicious versus legit requests. Sampling temperature is 0.2 and the sets are small, so read these as rough signals only.
Caveats
- These are small-model results. Coding is the main strength, and HumanEval is a narrow test of short Python functions. It is not a measure of general coding ability.
- Qwen's own published numbers use different prompts and full precision, so they differ from the numbers here. Only compare rows within this table.
How to run (llama.cpp)
llama-server -m ZY-S2-Q5_K_M.gguf -c 8192 --host 127.0.0.1 --port 8080
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Write a Python function that checks if a string is a palindrome."}],"temperature":0}'
Limitations
- Weak at math, multi-step reasoning, and agent/computer-use tasks compared with its coding skill.
- It can write insecure code (see the security rows). Review generated code before you run it.
- English and Python are the focus, and other languages are less tested.
- It is a 1.5B model, so expect factual mistakes.
License and credit
Released under Apache-2.0. Built on Qwen2.5-Coder-1.5B by the Qwen team (Alibaba Cloud), which is licensed under Apache-2.0. Comparison models are Qwen2.5-Coder-0.5B-Instruct and Qwen2.5-Coder-1.5B-Instruct (official Q5_K_M GGUFs).