Most of this README.md and the plots are generated by the Spark-X2.5-4B-IQ4_XS quant.

Spark-X2.5-4B — IMatrix Quantized GGUF Variants

Overview

This repository provides the imatrix quantized variants of Spark-X2.5-4B, a compact, general-purpose language model designed for conversation, writing, translation, reasoning, coding, tool use, and agentic workflows. The model uses a hybrid attention architecture, supports a native context length of up to 1M tokens, and covers more than 200 languages. The quantized variants in this repository were created following the imatrix methodology, preserving high quality while offering flexible deployment across a range of precision levels.

Quantized Variants

The table below lists each quantized variant, its file size, and its recommendation status.

Note: Recommendations are written by Mii Fanboy.

Recommended Quants (w/ KV Cache Q8_0/Q8_0) For Different Use Cases

Note: If you are using CPU, stick with Q quants. IQ quants would be much slower on CPU.

KLD Benchmark Summary

The table below reproduces the KLD benchmark results for all quantized variants. It aggregates the evaluation measurements across KLD, perplexity, and accuracy metrics for every variant.

Methodology

Imatrix Quantization Method

The imatrix quants were created by converting the base Spark-X2.5-4B model into a range of quantized variants, including the IQ3, Q3, IQ4, Q4, Q5, Q6, and Q8 series. Each quantized model is evaluated to verify its accuracy, quality, and overall performance before being included in this collection.

Dataset Creation

We used the bartowski-imatrix-v5-semantic calibration dataset as the input data. The dataset was split into two parts: a 70% imatrix calibration portion and a 30% KLD evaluation portion, using a fixed seed to ensure reproducibility. The 70% portion is used for imatrix quantization calibration, while the 30% portion is reserved for KL divergence evaluation.

KLD Measurement Method

For KLD measurements, we first generated the baseline logits from the original base Spark-X2.5-4B model and saved them as the reference. Then, for each quantized model, we computed the KL divergence between the model's outputs and the baseline logits using the perplexity tool. The KLD measurements were performed with a context length (ctx) of 4096, a batch of 2048, and an ub of 512. The evaluation results, including Mean KLD, Median KLD, and accuracy metrics such as Mean PPL and Same Top-1 %, were aggregated and saved into the kld_benchmark_summary table.

Local Deployment

The GGUF file can be used for local inference with Ollama and LM Studio. Spark-X2.5 support is provided by XHToken/llama.cpp, so the Quick Starts below use this compatible implementation.

Ollama Quick Start

Build

git clone https://github.com/XHToken/llama.cpp.git llama.cpp-spark
git clone https://github.com/ollama/ollama.git ollama-spark
cd ollama-spark
export OLLAMA_LLAMA_CPP_SOURCE="$(cd ../llama.cpp-spark && pwd)"
cmake -S . -B build
cmake --build build --parallel 8

Import the GGUF

Replace the model path below with the absolute path to the downloaded GGUF file:

printf 'FROM /absolute/path/to/Spark-X2.5-4B.gguf\n' > ./Modelfile.spark

Create and Run

Start the Ollama server in the first terminal:

./ollama serve

Open a second terminal in the same ollama-spark directory:

./ollama create Spark-X2.5-4B -f ./Modelfile.spark
./ollama run Spark-X2.5-4B --think=false

--think=false disables thinking mode for faster, direct responses.

LM Studio Quick Start

Build the Compatible llama.cpp Runtime

git clone https://github.com/XHToken/llama.cpp.git llama.cpp-spark
cd llama.cpp-spark
cmake -S . -B build
cmake --build build --parallel 8

Configure LM Studio

1. Close LM Studio.

2. Back up the selected LM Studio runtime directory:

   <LM_STUDIO_HOME>/extensions/backends/<selected-runtime>/
   

3. Copy the llama.cpp-spark build output into the selected runtime directory, replacing the existing runtime files.

4. Place Spark-X2.5-4B.gguf in:

   <LM_STUDIO_HOME>/models/<org>/<name>/
   

Example runtime directory on Apple Silicon:

./build/bin/* -> ~/.lmstudio/extensions/backends/llama.cpp-mac-arm64-apple-metal-advsimd-<version>/

Run

Open LM Studio, select the model under My Models, click Load, and start a new chat.

You can also use the lms CLI:

lms ls
lms load <model>
lms chat <model>

License

Released under the Apache License 2.0.