Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED — FP8 (vLLM)
Block-wise FP8 (E4M3) quantization of DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED, optimized for vLLM inference.
Key Features
- Full 262K context on a single RTX 3090 (24 GB)
- Vision (multimodal) support preserved
- Thinking/reasoning mode with
qwen3parser - Tool calling support
- ~11.3 GB model footprint (vs ~19 GB in BF16)
- FP8 KV cache compatible for additional memory savings
Quantization Details
| Parameter | Value |
|---|---|
| Method | FP8 E4M3 block-wise |
| Block size | [128, 128] |
| Scale format | weight_scale_inv (bfloat16) |
| Activation scheme | Dynamic |
| Max quantization error | 3.57% (relative) |
| Quantized layers | 200 (linear projections, MLP) |
| Kept in original precision | 560 (embeddings, norms, vision encoder, lm_head, DeltaNet-specific layers) |
Quantization was performed on GPU (CUDA) to ensure correct FP8 casting. The format matches lovedheart/Qwen3.5-9B-FP8 conventions.
Compatibility
| Backend | Status | Notes |
|---|---|---|
| vLLM | Tested | Marlin kernel (W8A16) on Ampere; native FP8 on Ada/Hopper |
| SGLang | Not tested | May work but not validated |
| llama.cpp | No | Use GGUF quantizations instead |
Tested Hardware
- RTX 3090 (24 GB) — Ampere, no native FP8, uses Marlin kernel for weight decompression
- Should work on RTX 4090, A100, H100, and other GPUs with 24+ GB VRAM
Quick Start (vLLM + Docker)
# docker-compose.yml
services:
qwen:
image: vllm/vllm-openai:nightly
ports:
- "8020:8000"
volumes:
- ./models/qwen3.5-9b-fp8:/models/qwen3.5-9b-fp8:ro
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
ipc: host
command: >
/models/qwen3.5-9b-fp8
--served-model-name qwen3.5-9b
--max-model-len 262144
--kv-cache-dtype fp8_e4m3
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--enable-prefix-caching
--gpu-memory-utilization 0.95
--max-num-seqs 2
--limit-mm-per-prompt '{"video":4,"image":8}'
restart: unless-stopped
docker compose up -d
# Test
curl http://localhost:8020/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5-9b",
"messages": [{"role": "user", "content": "What is 2+2?"}],
"max_tokens": 200
}'
VRAM Usage (RTX 3090)
| Component | Size |
|---|---|
| Model weights (FP8) | ~11.3 GB |
| KV cache (FP8) | ~10.9 GB |
| CUDA graphs + overhead | ~0.5 GB |
| Total | ~22.5 / 24 GB |
Model Lineage
- Qwen/Qwen3.5-9B — Base model (Apache 2.0)
- trohrbaugh/Qwen3.5-9B-heretic-v2 — Heretic uncensored fine-tune (Apache 2.0)
- DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED — Claude 4.6 thinking distillation (Apache 2.0)
- This repo — FP8 block-wise quantization for vLLM
Credits
- Qwen Team (Alibaba) for the Qwen3.5 architecture
- DavidAU for the fine-tuned model
- trohrbaugh for the Heretic uncensored base
- lovedheart for the Qwen3.5-9B-FP8 reference format
License
Apache 2.0 — same as all upstream models.