Mistral-Heretica-12B-GGUF
GGUF quantizations of mrcuddle/Mistral-Heretica-12B.
Original model: a Task Arithmetic merge of Mistral-Nemo-Instruct-2407, Lumimaid-v0.2-12B, and absolute-heresy — tuned for uncensored roleplay and creative writing. 12B params, Mistral architecture.
Quantized with llama.cpp (release b9890). Run these in LM Studio, Ollama, KoboldCpp, SillyTavern, or llama.cpp directly.
Download
| File | Quant | Size | Description |
|---|---|---|---|
| Q8_0 | Q8_0 | 12 GB | Maximum quality, near-lossless. Overkill for most. |
| Q6_K | Q6_K | 9.4 GB | Very high quality, effectively lossless. |
| Q5_K_M | Q5_K_M | 8.1 GB | High quality. Balanced pick. |
| Q4_K_M | Q4_K_M | 7.0 GB | Good quality, best size/speed tradeoff. Recommended default. |
F16 (23 GB) is the unquantized conversion — only needed if you want to re-quantize yourself.
Which file should I choose?
Pick the largest quant that fits in your RAM/VRAM with room to spare (leave ~1–2 GB for context).
- 8 GB RAM/VRAM → Q4_K_M
- 12 GB → Q5_K_M or Q6_K
- 16 GB+ → Q6_K or Q8_0
For roleplay/creative use, Q4_K_M is plenty — quality loss over Q8 is negligible in practice.
Prompt format
Mistral instruct format (primary):
<s>[INST] {prompt} [/INST]
Most front-ends (SillyTavern, LM Studio) apply this automatically when you select a Mistral / Mistral-Nemo template. The Lumimaid component was also trained with ChatML, so ChatML works too — but Mistral format is the safe default.
Recommended settings
Mistral-Nemo (this model's base) is temperature-sensitive — high temperature makes it incoherent. Keep it low:
| Setting | Value | Note |
|---|---|---|
| Temperature | 0.5 – 0.7 | Do not exceed ~1.0. Lower = more coherent, higher = more creative. |
| Min-P | 0.05 | Primary truncation. Leave Top-P/Top-K off if using Min-P. |
| Repetition penalty | 1.05 – 1.1 | Or use DRY (0.8 / 1.75 / 2) if your front-end supports it. |
| Context | up to 16k reliable | Nemo's trained context is 128k but quality degrades well before that. |
Start at temp 0.6 / min-p 0.05 and adjust from there.
Run it
llama.cpp:
llama-cli -m Mistral-Heretica-12B-GGUF-Q4_K_M.gguf --jinja -p "Write a short noir opening."
Download one file with the HF CLI (skip the rest):
hf download AnkitAI/Mistral-Heretica-12B-GGUF \
Mistral-Heretica-12B-GGUF-Q4_K_M.gguf --local-dir .
Ollama:
ollama run hf.co/AnkitAI/Mistral-Heretica-12B-GGUF:Q4_K_M
Verified
Q4_K_M smoke-tested: loads correctly, follows instructions (valid JSON output), and produces coherent creative prose. ~5–6 tok/s generation on Apple Silicon (M-series).
Support the Project
If this model is useful in your work, you can support independent research:
License
The base model declares no explicit license. This merge inherits terms from its source models — notably Lumimaid-v0.2-12B, which is CC-BY-NC-4.0 (non-commercial). Treat this model as non-commercial unless you verify otherwise with the original authors. These are quantizations only; all model rights belong to the original creators.
Credits
- Original model: mrcuddle
- Merge components: NeverSleep, MuXodious, Mistral AI
- Quantization tooling: llama.cpp
More models: ankitaglawe.com