Orca-4B-Agentic
Faster • Cheaper • Stronger for agentic & coding tasks
Optimized successor of the original Rhea-4B-Coding / Athenea line.
What changed for better price/performance
| Aspect | Original multi-pass Rhea | Rhea-4B-Agentic (this) |
|---|---|---|
| Reasoning style | Forced 3-pass (implement→review→final) | Single-pass + optional <think> |
| Context | 262k | 32 768 (covers real agent workloads) |
| Forced long outputs | min_new_tokens=1024 | Removed – answers as long as needed |
| Special tokens | Broken prefixes + many vision | Clean + lean (tools + thinking only) |
| Generation defaults | High temp / long forced | Tuned 0.4 / 0.9 for quality + speed |
| Agentic readiness | Good | Improved tool-calling template |
| Inference cost (VRAM/time) | Higher (long forced reasoning) | Significantly lower |
→ Same 4B base intelligence, noticeably faster and cheaper to run, better real-world agentic behavior because it is no longer forced into three full generations.
Why this is better for agentic tasks
- Standard
<think>...</think>for chain-of-thought (optional, model decides when useful) - Clean, reliable tool-calling format
- No artificial multi-pass overhead that multiplies latency and cost
- Strong coding + reasoning focus retained
- System prompt encourages precise, secure, efficient solutions
Technical specs
- Architecture: Qwen3ForCausalLM (4B)
- Context: 32 768 tokens
- Special tokens:
<|im_start|>,<|im_end|>,<think>,</think>, tool tags - Recommended quant: Q4_K_M / AWQ for best speed/quality
Recommended settings
{
"temperature": 0.4,
"top_p": 0.9,
"top_k": 30,
"repetition_penalty": 1.05,
"max_new_tokens": 8192
}
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "./Orca-2-4B-high"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
messages = [
{"role": "user", "content": "Write a secure Python function that validates JWT tokens and handles expiration gracefully."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048)
print(tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=False))
Deployment tips (max cheapness)
- vLLM / SGLang:
--max-model-len 32768 - llama.cpp: Q4_K_M or Q5_K_M
- Keep context ≤ 16k–24k in production for optimal speed/VRAM
Credits
- Base: Qwen3-4B + Athenea / Rhea coding lineage
- Optimized for agentic use, single-pass thinking, lower cost
Apache 2.0