GRAFT-10B-A2B
GRAFT-10B-A2B is a ~10B-parameter mixture-of-experts chat model from UltraLabs. Only about 2B parameters are active per token, so it generates text at roughly the speed of a 2B model while carrying the knowledge of a much bigger one.
It was made by shrinking Qwen3-30B-A3B-Instruct-2507 to a third of its size with GRAFT, our own model-shrinking method, and then healing, distilling and chat-tuning the result.
- Made by: UltraLabs (SmallAICreator)
- Format: GGUF,
GRAFT-10B-A2B-Q4_K_M.gguf(6.27 GB) - Architecture: Qwen3-MoE, 48 layers, 40 experts (5 active per token), hidden size 2048
- Chat template: ChatML with tool calling (embedded in the GGUF)
- Context: 32K tokens (perfect 60/60 needle-in-a-haystack recall all the way to 32K, see below)
- Language: English first
- License: Apache 2.0
Quick start
llama.cpp
# Chat in the terminal
llama-cli -hf SmallAICreator/GRAFT-10B-A2B-GGUF:Q4_K_M -cnv \
--temp 0.7 --top-p 0.8 --top-k 20 --repeat-penalty 1.05
# OpenAI-compatible server on http://localhost:8080
llama-server -hf SmallAICreator/GRAFT-10B-A2B-GGUF:Q4_K_M --jinja \
--temp 0.7 --top-p 0.8 --top-k 20 --repeat-penalty 1.05
Ollama
ollama run hf.co/SmallAICreator/GRAFT-10B-A2B-GGUF:Q4_K_M
Apps
Any GGUF app works: LM Studio, Jan and others. Download GRAFT-10B-A2B-Q4_K_M.gguf and load it; the chat
template is read from the file.
Hardware
| RAM | 8 GB free (16 GB total recommended) |
| CPU only | works; on a laptop i5, roughly 4-5 tokens/s writing and ~20 tokens/s reading prompts |
| GPU | much faster; offload as many layers as your VRAM allows |
Recommended settings
| Setting | Value |
|---|---|
| Temperature | 0.7 |
| Top-p | 0.8 |
| Top-k | 20 |
| Repeat penalty | 1.05 |
Use a system prompt. Without one, the model sometimes introduces itself with its base model's name. A one-line system prompt fixes that:
You are GRAFT-10B-A2B, a helpful AI assistant made by UltraLabs.
Prompt format
ChatML. Tool calls use the <tool_call> format, and the embedded template handles OpenAI-style tools.
<|im_start|>system
You are GRAFT-10B-A2B, a helpful AI assistant made by UltraLabs.<|im_end|>
<|im_start|>user
What is the capital of Burkina Faso?<|im_end|>
<|im_start|>assistant
Sample replies
Unedited greedy outputs from this exact Q4_K_M file:
Is it true that chickens lay burger patties?
No, it's not true that chickens lay burger patties. Chickens are birds that lay eggs, not patties. The idea of chickens laying burger patties is a humorous and fictional concept, often used in jokes or as a playful metaphor.
What is the capital of France?
The capital of France is Paris.
Write a Python function that reverses a string.
def reverse_string(s): return s[::-1](followed by a usage example, an explanation of slicing and a loop-based alternative)
It also handles rarer facts, such as the capitals of Burkina Faso (Ouagadougou) and Kazakhstan (Astana).
Long context: perfect recall up to 32K
Needle-in-a-haystack test: a random secret code ("The secret code for the vault is VELVET-8550.") is hidden inside real Wikipedia text, and the model is asked to find it. 4 context lengths × 5 needle depths × 3 different codes = 60 tests, greedy decoding, run on this exact Q4_K_M file with llama.cpp.
| Context | Start | 25% | 50% | 75% | End | Total |
|---|---|---|---|---|---|---|
| 4K | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 15/15 |
| 8K | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 15/15 |
| 16K | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 15/15 |
| 32K | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 15/15 |
60/60: every code returned exactly, at every depth, all the way to 32K tokens.
🎣 Go Fish Showdown: GRAFT-10B vs GPT-OSS-20B
GRAFT-10B played a full game of Go Fish against GPT-OSS-20B and won 9 books to 4.
| GRAFT-10B | GPT-OSS-20B | |
|---|---|---|
| Parameters | 10B | 20B |
| Reasoning | ❌ None, one pass per move | ✅ Reasoning enabled |
| Books won | 9 | 4 |
How it played out:
- The game was tied 4–4 at the midpoint. From there GRAFT won every remaining book.
- GRAFT started weak, asking for Jacks over and over. It then switched strategy mid-game and began asking for the ranks it already held three of, all without any chain-of-thought.
- GPT-OSS played a reasonable strategy but got stuck repeatedly asking for a 7 that was still in the pond.
Caveat: This was a single game, and GRAFT was dealt the stronger starting hand, so luck played a part. A larger batch of games with mirrored deals (each shuffle played twice with the hands swapped) is planned to separate skill from luck.
Half the parameters. No reasoning. Still won. 🐟
How close is it to the original?
During healing we measured how closely GRAFT-10B-A2B's next-token predictions match the 30B model it came from (KL divergence on held-out web, book, code and chat text; lower is closer). Dense Qwen3 models scored on the same text for reference:
| Model | KL to Qwen3-30B-A3B-Instruct-2507 |
|---|---|
| Qwen3-1.7B | 0.71 |
| Qwen3-4B-Instruct-2507 | 0.40 |
| Qwen3-8B | 0.39 |
| Qwen3-14B | 0.34 |
| GRAFT-10B-A2B (before chat-tuning) | ~0.31 |
This measures similarity to the original, not benchmark scores. Our model was also trained to match it, which makes this an easier test for us than for the Qwen models. Standard benchmarks are coming.
Limitations
- Coding is its weakest area. It plans and explains well, but it can mix up languages (e.g. Java syntax in a Kotlin file) or call APIs that don't exist. Always run and test generated code.
- It talks confidently. Check important facts and verify that an agent actually did what it says it did.
- Chat-tuned in English. Other languages work but are less polished.
Credits
- Base model: Qwen3-30B-A3B-Instruct-2507 by the Qwen team (Apache 2.0).
- Shrinking, healing and chat-tuning: UltraLabs (SmallAICreator).
- Runs on llama.cpp.
Also from UltraLabs: GRAFT-1B (under 1 GB, runs on phones) and the L-AI Android app for running GGUF models locally.