Instella-MoE-16B-A3B-Think GGUF
GGUF quantizations of AMD's Instella-MoE-16B-A3B-Think, converted and optimized for local inference with llama.cpp-compatible runtimes that support the Instella-MoE architecture.
Overview
This repository provides a full collection of GGUF quantizations for [amd/Instella-MoE-16B-A3B-Think](la-MoE-16B-A3B-Think.
Instella-MoE-16B-A3B-Think is a Mixture-of-Experts reasoning model featuring approximately 16B total parameters with ~3B active parameters per token, designed for efficient high-quality inference while maintaining strong reasoning, coding, and instruction-following capabilities.
These GGUF files were generated to enable:
- Local inference
- CPU deployment
- GPU-accelerated llama.cpp inference
- Edge and workstation deployments
- Quantized execution with reduced memory requirements
- Reasoning-focused workloads
⚠️ Important Compatibility Notice
Instella-MoE is not currently supported by upstream llama.cpp at the time these GGUFs were produced.
The model introduces architecture components beyond standard DeepSeek-V3 implementations, including:
- Gated Attention
- FarSkip dual-residual connections
These quantizations were generated using the community fork:
https://github.com/csabakecskemeti/llama.cpp
Branch:
instella-moe
This fork implements:
InstellaMoEForCausalLM- Gated attention runtime support
- FarSkip support
- GGUF export support for Instella-MoE
As a result, these files currently require:
llama.cpp (instella-moe branch)
or any future upstream release that merges full Instella-MoE support.
Model Details
| Property | Value |
|---|---|
| Model | Instella-MoE-16B-A3B-Think |
| Organization | AMD |
| Architecture | Instella-MoE |
| Family | DeepSeek-V3 Derived |
| Total Parameters | ~16B |
| Active Parameters | ~3B |
| Format | GGUF |
| Purpose | Reasoning, Coding, General Assistant Tasks |
| Quantization | Multiple GGUF Variants |
| Base Model | amd/Instella-MoE-16B-A3B-Think |
Available Quantizations
Standard Quants
Smallest
- Q2_K
Q3 Family
- Q3_K_S
- Q3_K_M
- Q3_K_L
Q4 Family
- Q4_0
- Q4_1
- Q4_K_S
- Q4_K_M
Q5 Family
- Q5_K_S
- Q5_K_M
High Quality
- Q6_K
- Q8_0
IQ Quants
Importance Matrix (Imatrix) optimized quantizations:
- IQ2_M
- IQ3_XXS
- IQ3_XS
- IQ3_M
- IQ4_XS
- IQ4_NL
These quantizations generally achieve superior quality-to-size ratios compared to traditional quant methods.
Recommended Quant
For Low RAM Systems
Q2_K
IQ2_M
Best Balance
Q4_K_M
IQ4_XS
High Quality
Q5_K_M
Q6_K
IQ4_NL
Maximum Quality
Q8_0
BF16
Example Usage
llama.cpp
./llama-cli \
-m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
-p "Explain mixture-of-experts architectures."
Server Mode
./llama-server \
-m Instella-MoE-16B-A3B-Think-Q4_K_M.gguf \
-c 32768
Quantization Methodology
The conversion pipeline follows:
Hugging Face Model
↓
Convert to BF16 GGUF
↓
Generate Imatrix
↓
Create Standard Quants
↓
Create IQ Quants
↓
Upload to Hugging Face
BF16 Conversion
The original model weights were converted directly into GGUF BF16 format using the Instella-MoE-enabled llama.cpp conversion tools.
Importance Matrix Generation
Importance matrix calibration was generated using:
Salesforce/wikitext
wikitext-2-raw-v1
A lightweight calibration dataset was used to optimize IQ quantization quality while remaining practical on constrained hardware.
IQ Quantization
IQ quant variants were produced using llama.cpp's importance-matrix-aware quantization pipeline.
Build Environment
These GGUFs were generated on a resource-constrained environment designed to maximize reproducibility.
System Constraints
- ~15 GB RAM
- No swap
- ~109 GB temporary storage
- 4 CPU cores
Because the BF16 GGUF is approximately:
~32 GB
the importance matrix was computed from a smaller intermediate quantization to avoid memory exhaustion while still producing high-quality IQ variants.
Repository Notes
Generation workflow includes:
- Automatic resume support
- Upload tracking
- Incremental quant generation
- Disk-space-aware cleanup
- Fault-tolerant upload recovery
Each quant is generated, uploaded, and safely removed locally before proceeding to the next file.
Prompt Format
Instella-MoE-16B-A3B-Think is an instruction-tuned reasoning model.
Typical usage:
User: Explain the difference between MoE and dense transformers.
Assistant:
For best results:
- Use clear instructions
- Allow sufficient context length
- Enable model reasoning when your frontend supports it
- Use lower temperatures for factual tasks
- Use higher temperatures for creative tasks
Performance Expectations
General guidance:
| Quant | Quality | Memory Usage |
|---|---|---|
| Q2_K | Lowest | Minimal |
| Q3_K_M | Good | Low |
| Q4_K_M | Very Good | Moderate |
| IQ4_XS | Excellent | Moderate |
| Q5_K_M | Excellent | Higher |
| Q6_K | Near BF16 | High |
| Q8_0 | Maximum | Very High |
| BF16 | Reference | Highest |
Actual results depend on:
- Prompt complexity
- Context length
- Hardware
- Backend implementation
- Future Instella-MoE runtime optimizations
Acknowledgements
Thanks to:
- AMD for releasing Instella-MoE-16B-A3B-Think
- The llama.cpp community
- @csabakecskemeti for the Instella-MoE llama.cpp implementation
- The GGUF ecosystem and local AI community
Disclaimer
This repository only provides GGUF conversions and quantizations.
Model behavior, weights, training methodology, benchmark performance, and intended use remain the responsibility of the original model authors.
Please refer to the upstream model card for official documentation:
👉 https://huggingface.co/amd/Instella-MoE-16B-A3B-Think
Download Stats Welcome ⭐
If these quantizations help your projects, research, benchmarking, or local AI deployments, consider liking the repository and sharing feedback.
Happy inferencing 🚀