Qwen3.6-35B-A3B-Hermes-V6 (APEX-Compact, GGUF)
A mixture-of-experts, multimodal (vision-language) model with a Hermes-style agentic / function-calling alignment, distributed as a compact GGUF for llama.cpp / llama-server.
In the author's local 8-model benchmark suite, this build placed #1 in every category — math, knowledge, and agentic tool-use — making it the top composite scorer of the set.
📊 The full interactive report is also bundled in this repo:
FINAL_REPORT.html.
✨ What this is
| Base lineage | Qwen 3.6 family (community build / fine-tune — not an official Alibaba release) |
| Architecture | Mixture-of-Experts (MoE), Qwen3 |
| Total / active params | ~35B total / ~3B active per token (the A3B notation) |
| Modality | Text + Vision (ships with a separate mmproj multimodal projector) |
| Alignment | Hermes V6 — agentic, function-calling, instruction-tuned |
| Format | GGUF — APEX-Compact mixed-precision (~4 bpw-class) |
| Main file size | ~17 GB |
| Serving | llama.cpp / llama-server (OpenAI-compatible API) |
The MoE design is the key to its efficiency: although ~35B parameters are loaded, only ~3B are active on any given token, so it runs at usable speeds on consumer APUs/GPUs while keeping dense-model quality.
📦 Files in this repository
| File | Size | Description |
|---|---|---|
Qwen3.6-35B-A3B-Hermes-V6-APEX-Compact.gguf | ~17 GB | Main model weights (APEX-Compact GGUF). Load this with llama-server -m. |
Qwen3.6-35B-A3B-Hermes-V6-mmproj.gguf | ~858 MB | Multimodal projector (vision encoder adapter). Load with --mmproj. Required for image input. |
FINAL_REPORT.html | ~13 KB | Interactive benchmark report (champion board + full breakdown). |
🏆 Benchmark results
Measured by the author on local hardware (see Methodology & caveats below). This model is 🥇 #1 in the 8-model comparison.
Composite champion board (top of the set)
| Model | Size | GSM8K (math) | Knowledge | Agentic | Composite | Speed |
|---|---|---|---|---|---|---|
| 🥇 Qwen3.6-35B-Hermes-V6 (this) | 16.2 GB | 91.0% | 78.1% | 90.0% | 86.4% | ~15 t/s |
| 🥈 Gemma-4-31B | 8.0 GB | 84.0% | 73.4% | 90.0% | 82.5% | 6.5 t/s |
| 🥉 Gemma-4-12B-QAT | 6.3 GB | 88.5% | 75.2% | 80.0% | 81.2% | 10.6 t/s |
Knowledge benchmarks — full breakdown (200 questions each, 2 runs averaged)
| Benchmark | Score |
|---|---|
| ARC-Challenge | 93.0% |
| HellaSwag | 85.5% |
| OpenBookQA | 94.5% |
| BoolQ | 89.5% |
| Winogrande | 75.0% |
| TruthfulQA | 31.0% |
| Knowledge average | 78.1% |
Math & agentic
- GSM8K: 91.0% — via
lm-evaluation-harness(industry standard). - Agentic: 90.0% (9/10) — τ-bench-style retail customer-service simulator (multi-turn, strict pass/fail): cancel orders, returns, refunds, exchanges, policy refusals, error recovery, and tool restraint.
⚠️ TruthfulQA is low for every model in this set (25–31%). TruthfulQA is adversarial by design; these subsampled, locally-judged scores are not directly comparable to official leaderboard figures.
🚀 How to run
Prerequisites
- A recent build of llama.cpp (CPU, Vulkan, or ROCm/CUDA backend).
- Enough RAM/VRAM for ~17 GB of weights (the MoE means compute is light, but all experts must reside in memory).
Serve it (OpenAI-compatible API, with vision)
llama-server \
-m Qwen3.6-35B-A3B-Hermes-V6-APEX-Compact.gguf \
--mmproj Qwen3.6-35B-A3B-Hermes-V6-mmproj.gguf \
-ngl 99 \
-c 32768 \
--port 8080 \
--host 0.0.0.0
Then talk to it like any OpenAI API at http://localhost:8080/v1/chat/completions.
Function calling / tool use
The Hermes V6 alignment supports the standard OpenAI tools schema (tool_choice: auto). Example:
curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "Qwen3.6-35B-A3B-Hermes-V6-APEX-Compact.gguf",
"messages": [{"role":"user","content":"Find my pending order and cancel only that one."}],
"tools": [ /* your function schemas */ ],
"tool_choice": "auto",
"temperature": 0.1
}'
Thinking mode control
This is a hybrid reasoning model. Prefix the system prompt with /no_think for fast, non-reasoning responses (recommended for high-volume tool-calling / agent loops); omit it to engage chain-of-thought reasoning.
Multimodal (image) input
With the --mmproj projector loaded, you can pass images (via the llama-server multimodal interface) alongside text for image-understanding tasks.
🔬 Methodology & caveats (please read)
These results are author-measured on a specific local setup, not official leaderboard scores. Be aware:
- Knowledge (MCQ): 200-question subsamples per dataset, averaged over 2 independent runs (matched within ±2.5%).
- GSM8K: run through
lm-evaluation-harness(EleutherAI) — the industry-standard harness. - Agentic: a custom τ-bench-style simulator (10 multi-turn retail tasks, strict pass/fail). The headline figure is the best of dedicated + loop runs, each model run with its optimal flags.
- Hardware: AMD Ryzen AI 9 HX 470 · Radeon 890M (~14.4 GB) · 28 GB RAM · Vulkan/ROCm backend. Speeds (~15 t/s) are specific to this device.
- Subsampled/local judging means absolute numbers are illustrative, not directly comparable to full-scale published benchmarks. Use them to rank models within this set, not to compare against vendor-published scores.
Full details and the interactive charts are in FINAL_REPORT.html.
⚖️ Licensing & attribution
- This is a community release by
MrFadiAi. It is not an official Alibaba / Qwen or Nous Research / Hermes product. - The
license: apache-2.0field follows the upstream Qwen3 lineage. Verify and comply with the applicable upstream Qwen license for your use case before redistribution or commercial deployment. - "Hermes" and "Qwen" are trademarks of their respective owners and are used here only to describe the model's training lineage.
🙏 Acknowledgements
Built on the outstanding work of the Qwen team, the Hermes alignment tradition, the llama.cpp project, and EleutherAI's lm-evaluation-harness.