Qwen3.5-4B-Unredacted-MAX - GGUF
GGUF quantized versions of prithivMLmods/Qwen3.5-4B-Unredacted-MAX, a 4.5 billion parameter language model based on the Qwen3.5-4B architecture, optimized for improved loading stability and compatibility with modern Transformers pipelines.
Model Details
- Base Model: Qwen/Qwen3.5-4B
- Fine-tuned by: prithivMLmods
- Architecture: Qwen2 (28 layers, 28 attention heads)
- Context Length: 32768 tokens
- Vocabulary Size: 151936
- Parameters: 4.5B
Quantization
| Filename | Bits | Size | Use Case |
|---|---|---|---|
model_f16.gguf | 16 | ~8.4 GB | Maximum quality, high VRAM requirement |
model_q8_0.gguf | 8 | ~4.5 GB | High quality, moderate VRAM |
model_q6_k.gguf | 6 | ~3.4 GB | Good quality, balanced VRAM |
model_q5_k_m.gguf | 5 | ~2.8 GB | Recommended for most use cases |
model_q5_k_s.gguf | 5 | ~2.5 GB | Compact, minimal quality loss |
model_q4_k_m.gguf | 4 | ~2.1 GB | Good balance, low VRAM |
model_q4_k_s.gguf | 4 | ~1.9 GB | Compact model |
model_q3_k_l.gguf | 3 | ~1.7 GB | Very compact |
model_q3_k_m.gguf | 3 | ~1.5 GB | Minimal size |
model_q3_k_s.gguf | 3 | ~1.4 GB | Extreme compression |
model_q2_k.gguf | 2 | ~1.1 GB | Maximum compression |
VRAM Requirements
| Quantization | VRAM | Recommended Device |
|---|---|---|
| F16 | 9 GB | High-end GPU |
| Q8_0 | 5 GB | Mid-range GPU |
| Q6_K | 3.5 GB | Mid-range GPU |
| Q5_K_M | 3 GB | Standard GPU |
| Q4_K_M | 2.5 GB | Standard GPU / Laptop GPU |
| Q3_K_M | 2 GB | Laptop GPU / CPU |
| Q2_K | 1.5 GB | CPU with sufficient RAM |
Usage
llama.cpp
./main -m model_q5_k_m.gguf -n 256 -p "You are a helpful assistant."
llama-cpp-python
from llama_cpp import Llama
llm = Llama(
model_path="model_q5_k_m.gguf",
n_ctx=32768,
n_threads=8
)
response = llm("What is machine learning?", max_tokens=256)
print(response["choices"][0]["text"])
LM Studio
Download the quantized GGUF file and load it in LM Studio's model picker.
Ollama
ollama pull tinyopsec/qwen3.5-4b-unredacted-max-gguf:q5_k_m
ollama run tinyopsec/qwen3.5-4b-unredacted-max-gguf:q5_k_m
Model Capabilities
This model excels at:
- Instruction Following: Optimized for direct response generation and multi-step instructions
- Reasoning Tasks: Capable of complex reasoning and problem-solving
- Text Generation: High-quality content generation across diverse topics
- Research & Experimentation: Designed for transformer behavior analysis and instruction dynamics understanding
- Lightweight Deployment: Efficient 4.5B parameter architecture suitable for local inference
License
This quantized version maintains the original model's Apache 2.0 license. See the original model card for full details.
Disclaimer
These are community quantizations. For official support and documentation, refer to the original model repository and llama.cpp documentation.