Kimi-K2.7-Code DSpark speculator

Overview

A DSpark speculator model for the Kimi-K2.7-Code base model, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel draft backbone with two lightweight heads: a Markov logit-bias head (low-rank intra-block token dependency) and a per-position confidence head (accept-rate prediction). This checkpoint was trained in the Camelot-Ray online pipeline, where the draft consumes hidden states streamed from a live Kimi-K2.7-Code vLLM server.

This export is from Camelot exp38 checkpoint 3.

Model Specifications

• Base Model: Kimi-K2.7-Code
• Format: Safetensors (single-file bf16, 6.3 GB, 44 tensors)
• Draft: 3 layers (Qwen3-style GQA), hidden 7168, 56 heads / 8 KV heads, head_dim 128, FFN 18432, rope_theta 50000, block_size=8
• Vocabulary: pruned draft vocab 32,000 (d2t/t2d remap tables shipped in the weights), target vocab 163,840; mappings reused from the Kimi-K2.6/K2.7 compatible tokenizer setup
• DSpark heads: Markov rank 256 (vanilla), confidence head (with-markov), mask_token_id=163608
• Aux hidden-state layers: [1, 29, 57]
• Trained context: seq 20000

Evaluation Results

Online vLLM nightly spec-decode, greedy decoding, TP=8, Kimi-K2.7-Code verifier, max_model_len=20000, cudagraphs enabled, and fuse_allreduce_rms=false.

The table also includes Novita's public Eagle3-MLA draft novita/kimi-k2.7-code-eagle3-mla under the same Kimi-K2.7-Code verifier, TP=8, cudagraph, and fusion-off serving setup. Cells show tok/s / speedup / accept_len. The standard rows use 6 prompts per benchmark; code-extra rows use the full LiveCodeBench and SPEED-Bench coding manifests with max_tokens=512.

Use DSpark with num_speculative_tokens=7 as the default for code, math, and most reasoning traffic.

Serving with vLLM

Requires a vLLM nightly with DSpark support:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly

vllm serve moonshotai/Kimi-K2.7-Code \
    --tensor-parallel-size 8 \
    --max-model-len 20000 \
    --trust-remote-code \
    --compilation-config='{"pass_config": {"fuse_allreduce_rms": false}}' \
    --speculative-config '{
        "model": "novita/kimi-k2.7-code-dspark",
        "num_speculative_tokens": 7,
        "method": "dspark"
    }'

Known vLLM-nightly caveats, with workarounds:

1. Draft-side FA3 AOT scheduling can crash with scheduler_metadata must have shape (metadata_size) because the GPU-worker spec-decode path misses fast_build=True when building draft attention metadata. Patch vllm/v1/worker/gpu/spec_decode/speculator.py and vllm/v1/worker/gpu/attn_utils.py to pass fast_build=True.
2. CUDA-graph capture can fail with a flashinfer allreduce workspace-size error under spec-decode token expansion; disable the fusion: --compilation-config='{"pass_config": {"fuse_allreduce_rms": false}}'.

Training Details

• Initialization: continued from the Kimi-K2.7-Code DSpark exp37 checkpoint
• Data: Kimi-K2.7-Code training mix with public Kimi-MTP data and hidden states streamed from the live Kimi-K2.7-Code verifier; seq 20000
• Steps: 20000 optimizer steps
• Schedule: lr 3e-4 cosine, warmup 300, global batch 8, accumulation 2
• Loss: 0.1 CE + 0.9 TV over block-diffusion anchors, decay_gamma 4.0, max_anchors 3072
• Semantics: apply_verifier_norm=False, hidden_states = concat of aux layers [1, 29, 57]