DeepSeek-V4-Flash-Vision-Exp-GGUF (Text-Only)

⚠️ Important: Text-Only Quantization

This repository contains text-only GGUF quantizations of DeepSeek-V4-Flash-Vision-Exp.

• ✅ Full text reasoning capabilities preserved
• ✅ All 7 quantization levels with PPL validation
• ✅ Calibrated with high-quality community dataset
• ❌ Vision encoder not included (image input not supported)

Vision support in GGUF format requires upstream llama.cpp integration, which is still in development. This release focuses on delivering production-ready text quantizations with comprehensive quality validation.

For multimodal capabilities, use the original safetensors model from deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.

Model Details

Quantization Quality

All quantizations calibrated with high-quality community dataset and validated with perplexity testing.

Baseline (BF16): PPL = 2.3549

Quality Notes

• Q3_K_M through Q2_K: Production-ready, minimal quality loss
• IQ2_XS through IQ2_XXS: Acceptable for most use cases
• IQ1_M and IQ1_S: Experimental, noticeable degradation

All models maintain structural integrity with 1328 tensors and 256 experts verified.

How to Use

Requirements

# Latest llama.cpp with DeepSeek-V4 support
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout master  # Ensure you have DeepSeek-V4 support
make LLAMA_CUDA=1    # or your preferred backend

Basic Inference

# Download a quantization (example: Q3_K_M, 130GB)
huggingface-cli download qtum/DeepSeek-V4-Flash-Vision-Exp-GGUF \
  --include "DeepSeek-V4-Flash-Vision-Exp-Q3_K_M-*" \
  --local-dir ./models

# Run inference
./llama-cli \
  -m models/DeepSeek-V4-Flash-Vision-Exp-Q3_K_M-00001-of-00004.gguf \
  -p "Explain quantum computing in simple terms:" \
  -n 512 \
  -ngl 99  # Offload to GPU

Server Mode

./llama-server \
  -m models/DeepSeek-V4-Flash-Vision-Exp-Q3_K_M-00001-of-00004.gguf \
  -c 4096 \
  -ngl 99 \
  --host 0.0.0.0 \
  --port 8080

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="models/DeepSeek-V4-Flash-Vision-Exp-Q3_K_M-00001-of-00004.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,  # Use GPU
    verbose=False
)

response = llm.create_chat_completion(
    messages=[
        {"role": "user", "content": "Write a Python function to calculate Fibonacci numbers."}
    ],
    max_tokens=512,
    temperature=0.7
)

print(response['choices'][0]['message']['content'])

Hardware Requirements

Estimates include model weights + KV cache. Actual requirements vary by context length.

Technical Details

Quantization Method

• Calibration Dataset: High-quality community dataset (MIT licensed)
• Protected Layers:
  • Attention: Q8_0
  • Shared Experts: Q8_0
  • Router/Indexer: Q8_0
  • Compressor: Q8_0
  • Hyper-Connections: F32
  • FFN Gate Input: F32
  • Embeddings/Output: Q6_K
• Verification: All quantizations validated with tensor count (1328) and expert count (256)

Architecture Highlights

• MoE Structure: 256 experts per layer, top-6 routing
• Active Parameters: 50.4B per forward pass
• Hash Routing: First 3 layers use hash-based expert selection
• Hyper-Connections: Cross-layer skip connections for improved training

Changelog

2026-09-04 - Initial release

• 7 quantization levels (Q3_K_M to IQ1_S)
• Text-only conversion with vision tensors filtered
• Comprehensive PPL validation
• Community calibration dataset

Limitations

1. No Vision Support: Image inputs are not supported. Text-only inference.
2. Large Memory Requirements: Even the smallest quant (IQ1_S) requires 61GB VRAM.
3. Split Files: Models are split into 4 shards due to size. Download all parts.

Citation

@misc{deepseek-v4-flash-vision-exp-gguf,
  title={DeepSeek-V4-Flash-Vision-Exp-GGUF (Text-Only)},
  author={qtum},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/qtum/DeepSeek-V4-Flash-Vision-Exp-GGUF}}
}

Original model by DeepSeek AI:

@misc{deepseek-v4-flash-vision-exp,
  title={DeepSeek-V4-Flash-Vision-Exp},
  author={DeepSeek AI},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp}}
}

License

This quantization is released under the MIT License, matching the base model.

About qtum

qtum publishes high-quality GGUF quantizations with transparent quality metrics and open calibration data. All releases include perplexity validation and are tested for structural integrity.

For questions or issues, please open an issue on this repository.