Sharp-Spark-X2.5-4B-GGUF

Dynamic imatrix GGUF quants of XHToken/Spark-X2.5-4B, a 4B dense long-context model, carrying our cyber-and-coding-weighted importance matrix, a heuristic per-tensor bit allocation, and the Sharp-Spark chat template.

This is a small, fast, long-context coder that fits 6 GB-VRAM GPUs and still holds usable speed out to six-figure context.

Who is this for?

This is the coder model for people with a small-VRAM GPU and ≤ 16GB RAM.

If you have more regular RAM than 16GB, try a (Cyber)TielCoder with partial GPU offloading instead. Its MoE architecture makes it fast even when it's split onto system RAM, and its about twice as capable.

Which quant should I download?

Grab the Q6_K_XL when you can: its performance is validated. The Q4 is a fallback.

If you find yourself needing a smaller quant than Q4, you should consider picking a natively smaller model instead. Small models like this are sensitive to quantization damage below Q6.

Long context

Spark is a hybrid-attention model: of its 36 layers, only 9 are full-attention (every 4th layer); the other 27 use a 512-token sliding window. So the KV cache barely grows — only the 9 full layers scale with context, which is what makes a 4B usable at six-figure context.

Short vs 131k context:

Decode fell only 2.6× going from empty to 131k depth. 42 tok/s at 131k and a ~4-minute full-131k prefill are genuinely usable.

KV cache cost. Only 9 of the 36 layers hold a full-length cache — the other 27 are windowed to 512 tokens and cost a flat ~57 MB, so KV grows at ≈ 4 KB / token / full layer, i.e. 36 KB/token at f16 and 19 KB/token at q8_0:

So on a 6 GB card, Q6_K_XL (3.6 GB) leaves room for ~48k at f16 — or ~96k at q8_0. For more still, drop the model a tier rather than the cache: Q4_K_XL clears 128k on the same card.

The imatrix

Quantizing means deciding which weights can afford to lose precision. An importance matrix answers that empirically instead of by guesswork: run the full-precision model over a calibration corpus, record how much each weight actually moves the output, and spend bits where the measurement says they matter.

This one is weighted toward security and systems code — a public, redistributable corpus of the kind of text this model is built to work on. It was measured on the BF16 weights directly: at 8.2 GB the full-precision model fits in memory, so no step in the measurement chain is itself an approximation of the model.

• 3641 chunks × 4096 tokens = 14.9 M tokens. The run consumed the corpus to exhaustion rather than stopping at a preset chunk budget, so the matrix reflects all of the calibration text, not a prefix of it.
• Every chunk is 4096 tokens, and that length is the whole point. Spark's attention is mostly windowed: 27 of its 36 layers only ever see 512 tokens at a time, and just 9 attend across the full sequence. Calibrate on chunks at or below that 512-token window and those 9 layers are never put into the long-range regime they exist for — their importance scores would describe work they aren't doing, and the quantizer would then under-protect exactly the layers that carry long-context behaviour. At 4096 they are measured doing their real job.

Both figures are recorded in the file itself as imatrix.chunk_count and imatrix.chunk_size, so you can check them rather than take our word for it.

The imatrix .gguf ships in this repo — point llama-quantize --imatrix ... at it and bake your own tier if none of ours is what you want.

The Sharp-Spark template

The embedded chat template is Spark's own, with four defects fixed and a terseness instruction spliced in. It keeps Spark's native control tokens and DeepSeek-style tool-call format untouched: this is Spark's own template with targeted repairs on top, not a template borrowed from another model.

Fixes over the stock template:

1. A served system prompt replaces the canned "you are a helpful assistant" default instead of trailing it.
2. The tools block is separated and ordered after the system prompt (stock ran them together and split the prompt around the tool list).
3. Reasoning history is read from reasoning / thinking as well as reasoning_content (vLLM / Responses-API compatibility).
4. Tool results render through the content-block path, so a list-of-blocks tool result no longer leaks a Python repr into the prompt.

Addition: a short terseness instruction is force-appended to the system prompt (opt out with chat_template_kwargs={"terse": false}). It also stands its tool block down if the runtime injects its own tool protocol (suppress_tool_instructions).

Terseness was active for the SWE-bench-Live board at the top of this card. That board sets this build against upstream as shipped, which moves the quant recipe and the template together — so it does not credit terseness with anything on its own. Flip the flag and A/B it if you need to know.

Tool calling works through llama-server --jinja, which parses Spark's tool-call syntax into standard OpenAI tool_calls (verified).

The raw template is in this repo as chat_template.jinja — apply it to your own quants of Spark-X2.5-4B with llama-server --chat-template-file chat_template.jinja, or embed it at quant time.

Requirements

Spark uses the spark2_5 architecture, which requires a recent llama.cpp with Spark support (the arch landed via PR #27868). An older build will fail to load these files. There is no MTP / draft head on this model, and no vision component.

Example:

llama-server -m Sharp-Spark-X2.5-4B-Q6_K_XL.gguf -ngl 99 -c 32768 --jinja \
  --temp 0.6 --top-p 0.95 --top-k 20

(temp 0.6 is our recommended operating point for coding.)

Provenance & license

• Base model: XHToken/Spark-X2.5-4B, Apache 2.0.
• BF16 GGUF converted from the base weights with the spark2_5-capable llama.cpp; each tier quantized from that BF16 with our imatrix under a heuristic per-tensor bit allocation.
• These quants inherit the base model's Apache 2.0 license.