DeepSeek-V3.2-Exp-AWQ

Base model: DeepSeek-V3.2-Exp

【Dependencies / Installation】

As of 2025-09-30, create a fresh Python environment and run:

pip install -U pip
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
pip install https://wheels.vllm.ai/dsv32/deep_gemm-2.1.0%2B594953a-cp312-cp312-linux_x86_64.whl

For more details, please refer to vLLM documentation [link]

【vLLM Startup Command - Single Node with 6 GPUs】

Note, as of 2025-09-30:

1. Only Hopper and Blackwell data center GPUs are supported for now.

2. The kernels are mainly optimized for TP=1, so it is recommended to run this model under DP/EP mode

3. DP mode may result in increased serving latency. To mitigate this, we recommend enabling MTP to maintain optimal speed. To disable MTP, simply remove the --speculative-config flag.

4. Some users have observed improved performance on H20 machines by setting export VLLM_USE_DEEP_GEMM=0

CONTEXT_LENGTH=32768
vllm serve \
    tclf90/DeepSeek-V3.2-Exp-AWQ \
    --served-model-name MY_MODEL \
    --data-parallel-size 6 \
    --enable-expert-parallel \
    --speculative-config '{"model": "tclf90/DeepSeek-V3.2-Exp-AWQ", "method": "deepseek_mtp", "num_speculative_tokens": 1}' \
    --enable-auto-tool-choice \
    --tool-call-parser deepseek_v31 \
    --swap-space 16 \
    --max-num-seqs 32 \
    --max-model-len $CONTEXT_LENGTH \
    --gpu-memory-utilization 0.99 \
    --trust-remote-code \
    --disable-log-requests \
    --host 0.0.0.0 \
    --port 8000

【Logs】

2025-09-30
1. Initial commit

【Model Files】

【Model Download】

from huggingface_hub import snapshot_download
snapshot_download('tclf90/DeepSeek-V3.2-Exp-AWQ', cache_dir="your_local_path")

【Overview】

DeepSeek-V3.2-Exp

Introduction

We are excited to announce the official release of DeepSeek-V3.2-Exp, an experimental version of our model. As an intermediate step toward our next-generation architecture, V3.2-Exp builds upon V3.1-Terminus by introducing DeepSeek Sparse Attention—a sparse attention mechanism designed to explore and validate optimizations for training and inference efficiency in long-context scenarios.

This experimental release represents our ongoing research into more efficient transformer architectures, particularly focusing on improving computational efficiency when processing extended text sequences.

• DeepSeek Sparse Attention (DSA) achieves fine-grained sparse attention for the first time, delivering substantial improvements in long-context training and inference efficiency while maintaining virtually identical model output quality.

• To rigorously evaluate the impact of introducing sparse attention, we deliberately aligned the training configurations of DeepSeek-V3.2-Exp with V3.1-Terminus. Across public benchmarks in various domains, DeepSeek-V3.2-Exp demonstrates performance on par with V3.1-Terminus.

How to Run Locally

HuggingFace

We provide an updated inference demo code in the inference folder to help the community quickly get started with our model and understand its architectural details.

First convert huggingface model weights to the the format required by our inference demo. Set MP to match your available GPU count:

cd inference
export EXPERTS=256
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}

Launch the interactive chat interface and start exploring DeepSeek's capabilities:

export CONFIG=config_671B_v3.2.json
torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --interactive

SGLang

Installation with Docker

# H200
docker pull lmsysorg/sglang:dsv32

# MI350
docker pull lmsysorg/sglang:dsv32-rocm

# NPUs
docker pull lmsysorg/sglang:dsv32-a2
docker pull lmsysorg/sglang:dsv32-a3

Launch Command

python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --page-size 64

vLLM

vLLM provides day-0 support of DeepSeek-V3.2-Exp. See the recipes for up-to-date details.

Open-Source Kernels

For TileLang kernels with better readability and research-purpose design, please refer to TileLang.

For high-performance CUDA kernels, indexer logit kernels (including paged versions) are available in DeepGEMM. Sparse attention kernels are released in FlashMLA.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2024deepseekv32,
      title={DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention}, 
      author={DeepSeek-AI},
      year={2025},
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.