Spark-X2.5-4B-GGUF

Llamacpp imatrix Quantizations of Spark-X2.5-4B by XHToken

Using XHToken/llama.cpp fork commit b10514-4a3635c32 (build tag b10514-4a3635c32) for quantization.

Original model: https://huggingface.co/XHToken/Spark-X2.5-4B

Companion repo: stornic56/Spark-X2.5-1.7B-GGUF — same pipeline, same calibration corpus, same fork commit. The family quality floor study below compares both.

Model details:

• Parameter count: 4.11B
• Input support: text
• imatrix: yes - details

How to run

Update: as of llama.cpp b10828,the spark2_5 architecture is supported in mainline llama.cpp PR #27868. These files load on anystock build b10828 or newer. The XHToken fork is only needed for older builds - How to run.

Prompt format

The chat template is embedded in every GGUF; run with --jinja and it is applied automatically. The rendered format is:

<｜start▁of▁sentence｜><|System|>
{system_prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|User|>
{prompt}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|Bot|>
<think>

The model is a thinking model by default: reasoning is emitted inside <think>...</think> before the final answer. The embedded template also supports tool calling, matching the upstream chat_template.jinja.

Don't know which to choose? Grab Spark-X2.5-4B-Q4_K_M.gguf (2.60GB) - usually a good mix of size and performance. Download instructions available here.

Available files:

Note: a Q2_K quant was also generated and tested, but excluded from this release after it produced repetition loops on the extended coherence test, while IQ2_M (a smaller file) completed the same test correctly. Evidence: reproducibility/q2_k_repetition_loops.log. Quants below IQ2_M (IQ2_XS, IQ1_M, Q1_0 class) were not produced; at this parameter count the quality loss is severe.

Downloading using the Hugging Face CLI

First, make sure you have the Hugging Face CLI installed:

pip install -U "huggingface_hub[cli]"

Download a specific file:

hf download stornic56/Spark-X2.5-4B-GGUF --include "Spark-X2.5-4B-Q4_K_M.gguf" --local-dir ./

How to run

These quants run with mainline llama.cpp release b10828 or newer, which added native spark2_5 support - installable in oneline via llama.app:

curl -LsSf https://llama.app/install.sh | sh
llama-server -hf stornic56/Spark-X2.5-4B-GGUF:Q4_K_M

Interactive chat (template is embedded, --jinja applies it):

./build/bin/llama-cli -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 8192 --jinja -cnv

OpenAI-compatible server:

./build/bin/llama-server -m Spark-X2.5-4B-Q4_K_M.gguf -ngl 99 -c 16384 --jinja

Recommended sampling from the upstream model card: temp 1.0, top_p 0.95, top_k -1. Use --temp 0 for deterministic output.

Other runtimes: Ollama and LM Studio can run these files, but only with builds backed by the same fork - see the "Ollama" and "LM Studio" sections of the official model card for the build steps.

Flash Attention

llama.cpp enables enables FlashAttention automatically for the sliding-window layers of spark2_5. No extra flag is needed.

imatrix

All quants below bf16 were made using the imatrix option, computed with llama-imatrix from this fork. The calibration corpus is the wikitext-2-raw-v1 train split (Salesforce/wikitext), written to disk verbatim and in order by reproducibility/calibration.py. The corpus file is included in this repo: calibration_data.txt.

The imatrix is available here: Spark-X2.5-4B-imatrix.gguf.

{
  "dataset": "Salesforce/wikitext",
  "config": "wikitext-2-raw-v1",
  "split": "train",
  "rendering": "raw text, no chat template, no special tokens",
  "chunk_size": 512,
  "chunks": 128,
  "tokens_consumed": 65536,
  "consumption_order": "sequential from file start (deterministic)",
  "threads": 8,
  "final_perplexity": "49.3694 +/- 1.09206",
  "imatrix_format": "GGUF (fork default)",
  "notes": "corpus is plain prose; a future revision may render tool-calling and reasoning conversations through the chat template, as done by other maintainers"
}

Reproducibility check: two independent imatrix runs over the same file with the same parameters produced identical final perplexity (49.3694 ± 1.09206). Run log: reproducibility/imatrix-run.log.

Which file should I choose?

The first thing to figure out is how big a model you can run. If you want the model running as fast as possible, fit the whole thing in VRAM: aim for a file 1-2GB smaller than your GPU's total VRAM, leaving room for the KV cache (this model's hybrid attention keeps the cache small, which helps). If you want maximum quality and can tolerate partial CPU offload, add your system RAM and VRAM together and pick a file 1-2GB smaller than that total.

Next, K-quant or I-quant? On NVIDIA (CUDA) and Apple silicon, I-quants (IQX_X) generally offer better quality per bit below Q4. On Intel Arc via Vulkan (Mesa), our measurements show the opposite for generation speed: K-quants are faster at equal bit-width, while I-quants win on file size - see the benchmarks below. Either way, both families were validated for coherence in this release.

Benchmarks

Measured with llama-bench from the same fork commit, Intel Arc B580 (Battlemage, Mesa Vulkan driver), -ngl 99 -t 4, 3 runs. Raw log: reproducibility/benchmarks_gpu.txt.

Prompt processing is roughly flat across quants (~2k t/s, compute-bound on the GPU's matrix cores). Long-context reading, Q4_K_M with the whole model in VRAM: pp32768 = 309 t/s (~6GB VRAM total including KV cache). The upstream 1M-token context was not validated in this release.

CPU reference readings (i3-12100F, 8 threads, single runs, not formal benchmarks): BF16 ≈ 4.8 t/s, Q4_K_M ≈ 13.4 t/s, Q3_K_M ≈ 15.6 t/s, IQ2_M ≈ 10.9 t/s generation.

Validation

• test-llama-archs -a spark2_5 from the fork: OK on CPU (NMSE 0.00e+00), Vulkan (8.58e-08) and meta buffers. Roundtrip: SKIP is expected for this architecture.
• Greedy decoding (--temp 0) produced token-identical output on CPU and Vulkan GPU, verified on Q4_K_M.
• Every published quant completed a 700-token coding coherence test at --temp 0; Q2_K failed with repetition loops and was excluded (log linked in the files table).

Reproducibility

Everything needed to rebuild these files bit-for-bit is in the repo:

Exact commands:

python convert_hf_to_gguf.py Spark-X2.5-4B --outfile Spark-X2.5-4B-bf16.gguf --outtype bf16

./build/bin/llama-imatrix \
  -m Spark-X2.5-4B-bf16.gguf \
  -f calibration_data.txt \
  -o Spark-X2.5-4B-imatrix.gguf \
  -c 512 --chunks 128 -t 8

./build/bin/llama-quantize --imatrix Spark-X2.5-4B-imatrix.gguf \
  Spark-X2.5-4B-bf16.gguf Spark-X2.5-4B-Q4_K_M.gguf Q4_K_M

ARM/AVX information

llama.cpp automatically repacks weights into an interleaved layout at load time for faster inference on ARM and AVX machines, covering Q4_0, IQ4_NL and most K-quants. No special quant choice is needed for CPU inference.

Credits

Thanks to the XHToken/SparkLLM team for releasing the model and maintaining the llama.cpp fork, and to bartowski for the quantization card format this repo follows.

License & attribution

The upstream model is released under Apache License 2.0; these derivative quantizations inherit it. Preserve upstream attribution when redistributing. This is an unofficial community release, not endorsed by XHToken/SparkLLM.