Huihui-Qwen3.5-9B-abliterated - GGUF
This repository contains GGUF format model files for huihui-ai/Huihui-Qwen3.5-9B-abliterated.
💾 Provided Files and Quantizations
We provide a spectrum of quantizations to suit different hardware capabilities. If you are unsure which to choose, Q4_K_M is the recommended standard for the best balance of speed, size, and reasoning retention.
| Filename | Quant Type | File Size | Recommended RAM/VRAM | Use Case |
|---|---|---|---|---|
Huihui-Qwen3.5-9B-abliterated-Q4_K_S.gguf | Q4_K_S | ~5.1 GB | 8 GB | Fastest inference, lowest memory footprint. |
Huihui-Qwen3.5-9B-abliterated-Q4_K_M.gguf | Q4_K_M | ~5.4 GB | 8 GB | Recommended. Best balance of quality and size. |
Huihui-Qwen3.5-9B-abliterated-Q5_K_S.gguf | Q5_K_S | ~6.1 GB | 10 GB | Slightly higher quality, needs more memory. |
Huihui-Qwen3.5-9B-abliterated-Q5_K_M.gguf | Q5_K_M | ~6.3 GB | 10 GB | Excellent reasoning retention for 12GB+ GPUs. |
Huihui-Qwen3.5-9B-abliterated-Q6_K.gguf | Q6_K | ~7.4 GB | 12 GB | Near-lossless quality. |
Huihui-Qwen3.5-9B-abliterated-Q8_0.gguf | Q8_0 | ~9.6 GB | 16 GB | Virtually indistinguishable from the F16 base model. |
🚀 How to Use
These GGUF files are fully compatible with popular local AI software, including:
- LM Studio
- Ollama
- Jan
- text-generation-webui (Oobabooga)
- llama.cpp
Prompt Format (ChatML)
Qwen models strictly use the ChatML format for prompting. Your system prompt and user inputs must be wrapped exactly like this:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Write a simple Python script.<|im_end|>
<|im_start|>assistant