Inkling-Small — Mixed Quant GGUF
Two independently audited mixed-precision variants of
thinkingmachines/Inkling-Small, source revision
8cc5877b44d343f88b92086aa1fb72897950f06a.
Choose one variant directory to download. Each contains all of its shards,
checksums, tokenizer/processor/config assets, model card and validation reports.
ds4-dfm-rs serves MQ85GB text on one DGX Spark. llama.cpp does not
implement this source-layout contract. The bench below is cold ds4-bench
with MTP off, not an HTTP serving or 64K result. MQ89 was not timed.
No K2, dots, or Qwen throughput number is reused here.
MQ85GB text throughput
Measured 2026-09-22 on ds4-dfm-rs@f3aef7c,
one GB10, nvidia-smi -lgc 300,2200, prompt promessi_sposi.txt, 8192 cold
prompt tokens and 64 greedy tokens, three fresh processes. Under load the
SM clock stayed at 2190–2197 MHz.
| Prefill tok/s | Decode tok/s | |
|---|---|---|
| Median | 452.58 | 13.07 |
| Warp rollback | 445.01 | 13.06 |
| Tile samples | 453.50 / 452.58 / 452.56 | 13.06 / 13.12 / 13.07 |
The warp row is the same binary with DS4_INKLING_NO_LOGIT_TILE=1.
The 8192-token frontier (200058 logits) matched that rollback bit-exactly
(max_abs=0) on the earlier unlocked run. Default prefill chunk is 2048.
DS4_INKLING_PREFILL_CHUNK=1024 and DS4_INKLING_NO_LOGIT_TILE=1 restore
the previous paths. Contexts above 8192, including 64K, were not measured.
Write-up: docs/inkling-optimization-2026-09-22.md.
| Variant directory | Bytes | Decimal GB | GiB | Shards |
|---|---|---|---|---|
| MQ89 | 95,446,595,200 | 95.446595 | 88.891569 | 6 |
| MQ85GB | 85,732,587,168 | 85.732587 | 79.844694 | 6 |
GB = 1,000,000,000 bytes; GiB = 1,073,741,824 bytes. MQ89 addresses the original 84–89 GiB weight budget. MQ85GB addresses the additional 84–86 GB budget, saving 9.714008 GB for additional runtime headroom. MTP is separate and excluded.
Download one variant
# MQ89
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ89/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ89 && sha256sum -c SHA256SUMS)
# MQ85GB
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ85GB/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ85GB && sha256sum -c SHA256SUMS)
Both retain all 888 main tensors, all 42 decoder layers, 256 experts in every
sparse layer, and complete native BF16 vision/audio ingress and attention.
general.architecture=inkling, inkling.tensor_layout=source-interleaved-v1,
GGUF quantization version 2. Gate/up interleaving and original tensor names/shapes
are unchanged. Read INKLING-CONTRACT.md before native work.
Precision differences
| Group | MQ89 | MQ85GB |
|---|---|---|
| Attention, vision/audio | Original BF16 | Original BF16 |
| L2 routed experts | BF16 | Q8_0 |
| Embedding/head, dense L0–1, shared experts | Q8_0 | Q8_0 |
| L3–39 routed gate/up | IQ2_XXS | IQ2_XXS |
| L3–39 routed down | Q2_K | IQ2_XS |
| L40 routed gate/up | Q4_K | Q3_K |
| L40 down and all L41 routed matrices | Q4_K | Q4_K |
| Router/norm/bias/scale/convolution/relative controls | Native | Native |
MQ85GB changes 40 precisions and preserves the other 848 tensor payloads bit-exactly. Its L2 Q8 payloads also match the separately published Q8 reference. All native payloads match original BF16/F32 source bytes. L2 Q8 is the deliberate large capacity tradeoff; this variant does not retain that official NVFP4 BF16 exclusion. Detailed audits and results are in each variant directory.
Calibration and fidelity
Both use the same measured multimodal imatrix: 1,663 exact processed inputs / 1,578,255 valid decoder tokens. The balanced primary pass has importance weight 1.0; a real structured reasoning/tool coverage supplement has weight 0.1. Effective text/code/image/audio shares are 45.215/21.433/23.817/9.535 percent. The public calibration dataset contains the exact inputs and complete source lineage/weighting details.
All L3–39 target experts were observed. L41 expert 123 remained unobserved; its ordinary Q4_K no-imatrix payloads are identical in both variants. No missing importance vector is fabricated. Shared calibration/ contains NPZ/DAT importance and coverage. reproduction/ and provenance/ record code, compiler/backend context and precise pins.
| Offline fidelity suite | MQ89 mean KL vs BF16 | MQ85GB mean KL vs BF16 | MQ89 top-1 agreement | MQ85GB top-1 agreement |
|---|---|---|---|---|
| short | 0.309541 | 0.382427 | 0.824 | 0.784 |
| extended | 0.133307 | 0.146076 | 0.918 | 0.920 |
These are small teacher-forced diagnostics (including real document/speech), not task benchmarks or guarantees of equivalent quality. Individual probes, direct variant comparisons and 32-token-budget generation checks are included. Extended media were excluded from calibration; text rows were excluded from the structured supplement, but overlap with historical primary text pools cannot be fully ruled out. Offline GGUF weights were dequantized to BF16 on B300 GPUs, so those timings/memory figures are not native quantized serving data.
The weight budget leaves room for caches and activations but does not prove that a particular context fits on GB10. Seven global-attention layers alone need about 28 GiB BF16 KV at 1,048,576 tokens, before local state, activations and OS memory. Actual admission and serving measurements remain necessary.
Related artifacts
The earlier MQ89 root-level paths remain available at historical revision
f091d3672dc69a1af77cad92b63cdeddc145a039; current main uses variant directories.
The MQ89 weight bytes and hashes did not change during this organization.
Source model is Apache-2.0; dataset sources retain their respective licenses.