Runtime: available. A complete, self-contained vLLM runtime for 2× DGX Spark (GB10,
sm_121) is published and validated — see Runtime. GitHub: https://github.com/jjang-ai/dots3-note-gb10-vllm · Image:ghcr.io/jjang-ai/dots3-note-gb10-vllm:sm121-cu130.
What this is
An NVFP4 requantization of dots-studio/dots3-note-prev,
targeted at NVIDIA Blackwell (RTX 50, GB10 / DGX Spark) via vLLM's native
modelopt_fp4 path.
- Routed experts → NVFP4 (E2M1, group-16, FP8-E4M3 block scales + FP32 global scale — modelopt format).
- Kept high-precision (BF16): MLA/DSA attention, shared expert, router gates, the dense layer 0, the MTP / NEXTN draft head (layer 46), and the full vision (MoE-ViT) + audio encoders. Multimodal and speculative decoding are preserved, not stripped.
hf_quant_config.json:quant_algo=NVFP4, 10 exclude patterns (attention / shared / dense-0 / MTP / vision / audio / lm_head).- ~174 GB on disk. Fits TP2 across two 121 GB GB10 nodes.
| Property | Value |
|---|---|
| Base | dots-studio/dots3-note-prev (Apache-2.0) |
| Total / active params | 280B / 16B |
| Experts | 256 routed + 1 shared, top-8 |
| Attention | 13 DSA (top-2048) + 33 SWA |
| MTP / DSpark | 1 shared NEXTN head, 1.13B (layer 46) — kept BF16 |
| Vision / Audio | MoE-ViT 7B (1.2B act) / dense 800M — kept BF16 |
| Context | up to 512K (DSA-native fp8_ds_mla paged KV) |
| Expert precision | NVFP4 (E2M1, g16) |
| Input / Output | text · image · video · audio / text |
Recommended generation config
Defaults tuned for agentic / tool-use / coding work (not plain instruct chat).
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="dots3-note",
messages=[{"role": "user", "content": "Refactor this module and add tests."}],
temperature=1.0, # upstream-recommended sampling
top_p=0.95,
max_tokens=8192, # give agentic/coding turns room
extra_body={
# Reasoning control (dots3 is a boolean thinking toggle, not tiered effort):
# enable_thinking=True → deliberate multi-step reasoning (hard reasoning / agent planning)
# enable_thinking=False → direct answer (low latency, simple tools)
"chat_template_kwargs": {"enable_thinking": True},
},
)
| Setting | Recommended | Notes |
|---|---|---|
temperature | 1.0 | upstream default; lower (0.2–0.6) for deterministic code edits |
top_p | 0.95 | |
enable_thinking | True for agents/hard reasoning, False for simple/latency-sensitive | dots3 has no separate low/med/high tiers — this boolean is the reasoning switch |
max_tokens | ≥4096 (8192 for coding) | thinking + tool loops need headroom |
| repetition penalty | none | not recommended for this model |
Agentic tool calling is native. When serving, enable --enable-auto-tool-choice --tool-call-parser dots; the chat template emits/parses <tool_call> blocks. Pass your tools=[...] schema as usual — no custom system prompt needed.
Multimodal (image · video · audio)
messages = [{"role": "user", "content": [
{"type": "video_url", "video_url": {"url": "https://.../clip.mp4"}},
{"type": "text", "text": "Describe the performance and what can be heard."},
]}]
- Video inputs carry their audio track — the model reasons over both jointly.
- Image / audio / video all route through the preserved (BF16) encoders; multimodal quality is unaffected by NVFP4 expert quantization.
- The
preprocessor_config.jsonfor vision/video/audio is included.
Runtime — available
A complete, self-contained vLLM runtime for 2× DGX Spark (GB10, sm_121, CUDA 13, aarch64) over TP2:
- GitHub: https://github.com/jjang-ai/dots3-note-gb10-vllm
- Docker:
ghcr.io/jjang-ai/dots3-note-gb10-vllm:sm121-cu130
DeepSeek Sparse Attention (DSA) is not officially supported on consumer / GB10 Blackwell (sm_120/sm_121) in upstream vLLM — the fused sparse-attention and top-k CUDA kernels are Hopper / datacenter-Blackwell only and crash or emit garbage on GB10. This runtime carries the patches that make it correct and fast:
- DSA sparse-MLA decode + indexer → exact torch references (the deep_gemm / FlashInfer sm120 kernels are numerically wrong on GB10 across a KV-block boundary).
- DSA top-k →
torch.topkon sm_121 (the fused kernel needs ≥128 KB shared memory; GB10 exposes ~99 KB, so it crashes for any largemax_model_len). Mirrors upstream vLLM PR #49897. - NVFP4 via Marlin W4A16 (the FlashInfer / CUTLASS FP4 GEMMs no-op on sm_121).
- CUDA graphs (
FULL_AND_PIECEWISE) for ~1.8–2× decode; NCCL 2.30.4 preload to fix a dual-Spark deadlock;--skip-mm-profilingso multimodal and a large KV pool coexist.
Full root-cause writeup: docs/GB10_FIXES.md.
Validated performance (2× GB10, driver 580.x, TP2)
| Metric | Value |
|---|---|
| Decode (MTP + CUDA graphs) | ~24–25 tok/s (vs ~14 eager) |
| Prefill | >1000 tok/s past ~330k ctx (prefix reuse); decode ~flat in context length |
| Max coherent context | 428,671 tokens — single conversation, 0 errors (KV-pool bound) |
| Multimodal | image · video · audio all verified |
| Reasoning | default on; enable_thinking toggle per request |
Serve (Docker, both nodes)
docker pull ghcr.io/jjang-ai/dots3-note-gb10-vllm:sm121-cu130
# worker on spark2 FIRST, then head on spark1 (host networking; set your node IPs + RoCE iface)
IMAGE=ghcr.io/jjang-ai/dots3-note-gb10-vllm:sm121-cu130 MODEL_DIR=/path/to/dots3-note-prev-NVFP4 \
HEAD_IP=10.99.0.1 WORKER_IP=10.99.0.2 bash docker/run_worker.sh # spark2
IMAGE=ghcr.io/jjang-ai/dots3-note-gb10-vllm:sm121-cu130 MODEL_DIR=/path/to/dots3-note-prev-NVFP4 \
HEAD_IP=10.99.0.1 WORKER_IP=10.99.0.2 bash docker/run_head.sh # spark1
The container defaults to the fast config (MTP + CUDA graphs + NCCL preload). Every env knob and the
from-source patch (patches/apply.sh, no image needed) are documented in the GitHub repo.
Access
Open weights — no gating. Apache-2.0, inherited from upstream
dots-studio/dots3-note-prev.
Credits
- Base model: dots-studio —
dots3-note-prev. - NVFP4 requantization + GB10/TP2 runtime: JANGQ-AI.