Bible Assistant Qwen3.5-4B GGUF
GGUF quantized versions of Ttimms/Bible-Assistant-Qwen3.5-4B-v2.
Quantization
| Quant | Bits | Size | Use Case |
|---|---|---|---|
| Q8_0 | 8 | ~4.2 GB | Maximum quality, slower |
| Q6_K | 6 | ~3.1 GB | High quality |
| Q5_K_M | 5 | ~2.6 GB | Balanced quality/speed |
| Q5_K_S | 5 | ~2.4 GB | Balanced quality/speed |
| Q4_K_M | 4 | ~2.0 GB | Good quality/speed trade-off |
| Q4_K_S | 4 | ~1.9 GB | Good quality/speed trade-off |
| Q3_K_L | 3 | ~1.6 GB | Small, acceptable quality |
| Q3_K_M | 3 | ~1.5 GB | Small, acceptable quality |
| Q3_K_S | 3 | ~1.4 GB | Small, acceptable quality |
| Q2_K | 2 | ~1.1 GB | Minimal, fast |
VRAM Requirements
| Quantization | VRAM | Notes |
|---|---|---|
| F16 | ~8 GB | Full precision |
| Q8_0 | ~4.5 GB | Minimal overhead |
| Q6_K | ~3.5 GB | Minimal overhead |
| Q5_K_M | ~3.0 GB | Minimal overhead |
| Q4_K_M | ~2.5 GB | Minimal overhead |
| Q3_K_M | ~2.0 GB | Minimal overhead |
| Q2_K | ~1.5 GB | Minimal overhead |
Usage
llama.cpp
./main -m model_q5_k_m.gguf -p "Your prompt here" -n 256
llama-cpp-python
from llama_cpp import Llama
llm = Llama(model_path="model_q5_k_m.gguf", n_gpu_layers=-1)
output = llm("Your prompt here", max_tokens=256)
print(output["choices"][0]["text"])
LM Studio
Load any .gguf file directly in LM Studio.
Ollama
ollama create bible-assistant-qwen35 -f Modelfile
Where Modelfile contains:
FROM model_q5_k_m.gguf
PARAMETER temperature 0.7
Original Model
Ttimms/Bible-Assistant-Qwen3.5-4B-v2 — A 4B parameter model fine-tuned for Bible-related assistance and knowledge.
License
Refer to the original model's license.