⚡ Qwen3.8-Flash-Coder-26GB-GGUF (160 Experts Hardware-Aligned Coding Subnet)
This repository provides GGUF quantized formats of Jab1718/qwen3.8-flash-coder-85gb-bf16, a 160-expert coding subnet sliced from Qwen3.8-Flash-Next (335GB) using the moe-slice toolkit.
📦 Provided GGUF Files
| File Name | Quantization | Size on Disk | Target Hardware | Recommended Use Case |
|---|---|---|---|---|
qwen3.8-flash-coder-26gb-q4_k_m.gguf | Q4_K_M | 26.43 GB (27 GB) | 1x 24GB GPU (RTX 3090/4090), RTX 5000 Ada (32GB), or 32GB Mac | 🔥 92% VRAM reduction from 335GB base! Run locally on single consumer GPU! |
qwen3.8-flash-coder-43gb-q8_0.gguf | Q8_0 | 42.25 GB (43 GB) | 2x 24GB GPUs / 1x 48GB GPU / 64GB+ RAM Mac | Maximum reasoning fidelity (~99.5% BF16) |
Modelfile | Text config | <1 KB | Ollama | Quick import and deployment in Ollama |
🚀 Quick Start with Ollama
Step 1: Download the GGUF file and Modelfile
huggingface-cli download Jab1718/qwen3.8-flash-coder-26gb-gguf qwen3.8-flash-coder-26gb-q4_k_m.gguf --local-dir .
huggingface-cli download Jab1718/qwen3.8-flash-coder-26gb-gguf Modelfile --local-dir .
Step 2: Create and run model in Ollama
ollama create qwen3.8-flash-coder -f Modelfile
ollama run qwen3.8-flash-coder "Implement a thread-safe LRU Cache with TTL expiration in Rust."
🍏 Apple Silicon & Mac Setup Notes (Metal Acceleration)
[!IMPORTANT] Runtime Compatibility Notice for macOS Users:
- Ollama & Legacy Runtimes: Bundled Ollama versions (e.g., v0.32.5 or earlier) and older DS4 binaries hardcode
expert_count=512and will reject this file (expected expert_count=512 … got 160).- Recommended Runtime: Build and run using the latest
llama.cpp masterwith Metal support (which includes native compatibility PRs for arbitrary MoE expert counts likeqwen4exp).
⚡ Verified Apple Silicon Performance Profile (Tested on Apple M5 Pro, 64 GB Unified Memory, macOS 26)
- Extraordinary Memory Efficiency: Loads at full 64K context (
-c 65536) using only ~33 GB wired RAM (~46% RAM free) — massive headroom for concurrent developer workflows on 64 GB Macs! - Blazing Fast Decode: ~32 tokens/second sustained decode throughput.
- Near-Instant Initialization: Cold load in ~10 seconds via memory-mapped I/O.
Recommended llama-server Launch Command:
./llama-server \
-m ./qwen3.8-flash-coder-26gb-q4_k_m.gguf \
-c 65536 \
-ngl 999 \
--host 0.0.0.0 \
--port 8080
💡 Prompting Guidelines & Best Practices
1. Code Review & Bug-Finding Prompts (Preventing Empty Content)
- The Symptom: In analytical/code-auditing tasks (e.g., "Find all bugs in this function"), certain frontends/chat templates may capture reasoning inside
<think>...</think>but route near-empty text (0–2 chars) into the finalcontentfield upon encounteringfinish_reason=stop. - Root Cause: Chat template stop-token routing or missing explicit conclusion markers outside the thought block.
- Solution (Explicit Output Routing): Instruct the model explicitly to provide its structured audit report outside the thought tags:
# Recommended Prompt Pattern:
Review the following code for concurrency bugs and memory leaks.
Think thoroughly through edge cases inside <think>...</think>.
CRITICAL: After closing the </think> tag, you MUST provide your final structured bug report (specifying Bug Name, Line Number, Severity, and Recommended Fix) directly in your response content.
2. Surgical Code Refactoring (Preventing Over-Rewrites)
- To prevent the model from rewriting entire algorithms from scratch (which can introduce edge-case regressions like off-by-one errors):
- Best Practice: Prompt the model for a minimal diff or surgical patch:
# Recommended Refactoring Prompt:
Refactor the following function to fix the infinite loop bug.
Provide ONLY a minimal surgical unified diff patch (+/- lines) without rewriting unaffected logic.
3. Domain Focus & General Math Trade-off
- Note on Math/GSM8K: This 160-expert subnet was intentionally extracted by prioritizing Systems Programming, Algorithmic DP, and Tool-Calling trajectories. General word-problem arithmetic (GSM8K) was pruned by design to give maximum capacity to multi-language software development.
📊 Benchmark Capabilities
- Rust (Systems): 10/10 (100.0%) Pass@1
- C++20 (Modern Systems): 10/10 (100.0%) Pass@1
- TypeScript (Fullstack): 4/5 (80.0%) Pass@1
- Coding Agent (Diff/FIM/Bugfix): 16/20 (80.0%) Pass@1
- Python Algorithms: 39/50 (78.0%) Pass@1
📜 Citation
@software{thainq2026moeslice_gguf,
author = {Thai Nguyen},
title = {Qwen3.8-Flash-Coder-GGUF: Hardware-Aligned Coding Subnet for llama.cpp and Ollama},
url = {https://github.com/Jab1718/Moe-slices},
year = {2026}
}