Solar Open2 250B Mixed-Quant GGUF
Built with Solar.
This repository contains the locked MXQ-v1 candidate derived from
upstage/Solar-Open2-250B.
It preserves the complete topology: 48 layers, all 320 routed experts per
layer, one shared expert, and the original [GQA, KDA, KDA, KDA] × 12
schedule. No expert, layer, or parameter pruning, merging, or distillation was
performed. This is an independent conversion and is not an official Upstage
release.
Verified throughput (2026-09-14, Rust host, 300–2200 MHz cap): On one
DGX Spark GB10, SM clock 2190 MHz. MXQ-v1 with K-FP8/V-FP4 KV. Cold
ds4-bench, Promessi Sposi, 4,096-token chunks, 64 greedy tokens, MTP
off. Interleaved medians of three; 196,608 frontier logits and 64 IDs
byte-identical.
Round 1 default-on FATTN_WS (DS4_SOLAR_FATTN_WS=0 restores pair):
| Prompt tokens | Prefill off → on | Decode off → on |
|---|---|---|
| 8,192 | 1,050.86 → 1,075.76 tok/s | 17.40 → 17.44 |
| 65,536 | 731.24 → 927.50 tok/s | 13.06 → 13.02 |
Round 2 skip Q3 handoff down sanitize (DS4_CUDA_MOE_HANDOFF_SANITIZE=1
restores the pass), measured with WS on:
| Prompt tokens | Prefill off → on | Decode off → on |
|---|---|---|
| 8,192 | 1,073.59 → 1,095.61 tok/s | 17.43 → 17.43 |
| 65,536 | 925.19 → 943.18 tok/s | 13.02 → 13.01 |
Runtime:
ds4-dfm-rs@67de331.
Disk-KV restart and HTTP partial fork now reuse prefixes. This is not HTTP
and not the uncapped September 7/12 benches. Keep the SM cap on GB10.
Protocol, serving, rejects.
The September 7 fused-MoE pass and September 12 uncapped controls use other clocks and are not this graph. 1,048,576-token serving is not claimed.
Support my work
I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.
Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.
Artifact
| Variant | Split | Size | Approximate BPW |
|---|---|---|---|
| MXQ-v1 | 11 shards | 95,533,532,160 bytes (88.973 GiB) | 3.05 |
Download the complete 11-file split beginning with
Solar-Open2-250B-MXQ-v1-00001-of-00011.gguf. Exact hashes are in
MXQ-v1-SHA256SUMS.
The full BF16 and full-model Q8_0 reference conversions are published
separately in
Baekpica/Solar-Open2-250B-GGUF.
Precision recipe
| Module family | Precision |
|---|---|
| Token embedding, LM head | Q8_0 |
| GQA q/k/v/o/g projections | Q8_0 |
| KDA matrix projections | Q8_0 |
| Shared expert gate/up/down | Q8_0 |
| Routed gate/up, layers 4–43 | IQ2_XXS + imatrix |
| Routed down, layers 4–43 | Q3_K + imatrix |
| Routed gate/up/down, layers 0–3 and 44–47 | Q4_K + imatrix |
| Router, norms, small KDA control/norm/conv tensors | F32 |
The final GGUF contains 1,083 tensors: 409 F32, 530 Q8_0, 80 IQ2_XXS,
40 Q3_K, and 24 Q4_K tensors. quant-recipe-v1.yaml and
verify-v1.json contain the machine-readable recipe and verification result.
Calibration
The Solar chat-template calibration corpus follows the published mix from
Baekpica/Solar-Open2-120B-A15B-REAM-148E-Healing-Mix: instruction chat 22%,
Cascade stage-1 reasoning 16%, Cascade stage-2 reasoning 16%, Korean 16%,
other multilingual 12%, finance 6%, SWE 6%, and algorithmic code 6%.
The imatrix processed 590 × 512-token chunks. Layers 1–47 reached strict
320/320 routed-expert coverage. Layer 0 retained one explicit, pinned
exception of 29 zero-activation experts; the exact IDs and gate are recorded
in verify-imatrix.json. No precision was lowered to compensate for this
exception.
Validation status
- All 11 shards passed split ordering, bounds, topology, tensor count, tensor family, and exact tensor-type verification with zero errors.
- The artifact loaded resident across four H100 NVL GPUs and completed 32 deterministic short generations plus eight longer no-think quality checks.
- Every candidate output was non-empty with valid token IDs. Fatal/content checks passed for Korean, English, Japanese, code, arithmetic, exact JSON, and constrained-list prompts; no NaN or malformed structured output was observed.
- A development ds4 CUDA runtime mapped the complete artifact on one H100 and passed full-weight, recurrent-state snapshot/restore, replay, and cold invalidation regressions.
- A native
sm_121ads4 build serves the complete artifact on one DGX Spark GB10 through the OpenAI-compatible server at-c 196608with three persistent banks. See DGX Spark for the measured point.
H100 timings are development and artifact-sanity evidence only. Do not present them as DGX Spark performance. 1,048,576-token serving on Spark is not claimed.
DGX Spark
One NVIDIA DGX Spark GB10 (128 GB unified memory, driver 610.43.02, CUDA
13.3, sm_121a). Runtime
Baekpica/ds4@b2e52b9
on the dfm line. Artifact: this MXQ-v1 11-shard split. A long-lived VMM
weight owner (--backend vmm --scope base --reserve-gb 16, 453 derived
aligned artifacts) plus a restartable ds4-server worker:
DS4_CUDA_WEIGHT_IPC_MANIFEST="$RUN/weights.manifest" \
DS4_CUDA_WEIGHT_IPC_SCOPE=base \
DS4_SERVER_COALESCE_MAX=3 \
DS4_SERVER_COALESCE_MAX_TOKENS=4096 \
DS4_CONT_PREFILL_CHUNK=4096 \
DS4_METAL_PREFILL_CHUNK=4096 \
./ds4-server -m Solar-Open2-250B-MXQ-v1-00001-of-00011.gguf \
--cuda -c 196608 -n 1024 -t 6 \
--host 0.0.0.0 --port 8002 --model-id solar-open2-250b \
--no-spec --no-update-check --mem-floor-gb 8
OpenAI Chat Completions, thinking disabled, exact-cold (cached_tokens=0),
128 decode tokens, median of three. Loaded SM clocks 2,411–2,561 MHz
(uncapped; not the 300–2200 MHz campaign above).
banks_total=3 still admitted.
| Depth | Prompt tokens | Prefill | Decode p50 | Decode API |
|---|---|---|---|---|
| 8K | 8,222 | 1,050.7 tok/s | 19.05 tok/s | 18.9 tok/s |
| 64K | 66,761 | 804.5 tok/s | 13.07 tok/s | 14.1 tok/s |
On the same host and artifact, before the dfm default-path series that
ends at b2e52b9, 8K decode was 17.5 tok/s and 64K average prefill was
710 tok/s. The series kept the production path as: skip width-1 inner
CUDA graph capture; walk FATTN KV in 32-key then GQA-pair 64-key tiles;
keep the KDA 64 KiB state tile in shared memory; tile grouped GQA decode
KV in 32 keys with packed K-FP8/V-FP4 dequant.
test-solar-kv on this binary: 512-token GQA2 vs one-head rel_rms=0;
split vs direct rel_rms=8.45e-7. Incremental T(64K)−T(60K) last-4K is
not a published figure. Native 1,048,576-token metadata is not a measured
Spark pass.
Partial prefix reuse
Runtime
Baekpica/ds4@042bcea
adds a bounded, shared Solar KDA checkpoint pool to the continuous lane.
The 32 slots are demand-mapped, exact forks share references, and long
prefills retain roughly 24 periodic checkpoints across the configured
context in addition to request boundaries. A divergent prompt restores the
nearest safe checkpoint and replays only the gap.
One controlled A/B run used the same GB10, artifact, four-bank worker,
-c 65536, 4,096-token prefill chunks, and greedy no-think Chat requests.
The control set DS4_SERVER_FORK_PARTIAL=0, which also disables checkpoint
reservation. The treatment used the default.
| Request | Prompt | Control cached | Treatment cached | Control TTFT | Treatment TTFT | Speedup |
|---|---|---|---|---|---|---|
| Source | 12,123 | 0 | 0 | 10,654.0 ms | 10,710.7 ms | 0.995× |
| Branch at ~6K | 6,079 | 0 | 4,096 | 5,323.7 ms | 1,868.8 ms | 2.85× |
| Branch at ~10K | 10,148 | 0 | 8,192 | 8,920.2 ms | 1,929.8 ms | 4.62× |
Both branch responses were byte-identical between control and treatment. Treatment reported two partial-fork admits, 12,288 cached tokens, zero failed requests, and zero admission rejects. Source checkpoint capture added 56.7 ms (0.53%) in this single run. After the three requests, device-live memory was 102.04 GiB and system-available memory was 14.59 GiB with an 8 GiB floor.
This is a bounded worker-local cache, not an unbounded radix tree. Token history and GQA prefix rows remain anchored to a retained bank, and the KDA checkpoint pool is not serialized into disk-KV payloads.
On 2026-09-14, a one-bank worker at -c 8192 with --kv-disk-dir and
partial fork, under the 300–2200 MHz cap, reused cached_tokens=538 after
a clean restart continuation and cached_tokens=4096 on a 4,177-token
divergent Chat (81 computed, TTFT 416 ms vs 4.7 s). Prompts shorter than
the 4,096-token stride still have no interior checkpoint.
Provenance
- Source:
upstage/Solar-Open2-250B@9190fbe63a2ad8e17fc766ccceb36de7c66f004b - Official implementation oracle:
upstageAI/transformers@ef5c37485091057070b36dd0620798f74988eb01 - llama.cpp conversion base:
ggml-org/llama.cpp@6ea215d171fd31df943bf1ac8227129f2b963160 - Solar conversion/runtime patch SHA-256:
7c267850de7a08af66a9bd64e9d4a3b33e085a4d2888d94a4c3d919f87283e44 - Reproduction repository:
Baekpica/solar-open2-mixed-ds4
Current upstream runtimes may not yet recognize the solar-open2 GGUF
architecture. Use a compatible Solar Open 2 llama.cpp port or the linked ds4
development branch; compatibility claims should be tied to an exact runtime
revision.
License and attribution
The source model and this derivative artifact are distributed under the
included Upstage Solar License. The model name begins with Solar, and this
card prominently states “Built with Solar” as required by that license.