Inkling-Small — intermediate GGUF artifacts
Intermediate artifacts for Inkling-Small Mixed Quant.
Independent conversion from thinkingmachines/Inkling-Small revision
8cc5877b44d343f88b92086aa1fb72897950f06a.
MTP-BF16 sidecar verified with multimodal MQ85GB serving (2026-09-10): Paired with the separate MQ85GB main artifact, ds4-dfm-rs on one DGX Spark serves text, PNG/JPEG image and 16 kHz WAV audio input with text output and checks this eight-layer draft stack on and off over HTTP. The Q8_0 main artifact in this repository is a structurally verified conversion and is not qualified for native serving. HTTP evidence runtime:
ds4-dfm-rs@4c32556, PR #28.
The MTP-BF16 sidecar is used with the separate MQ85GB main artifact in ds4-dfm-rs's DGX Spark integration. All eight draft layers have short CLI and serial HTTP MTP on/off checks. This does not qualify the 262.704 GiB Q8_0 main artifact for native serving. The separate GB10 performance report (PR #30) applies to MQ85GB with MTP off; it does not establish Q8_0-main throughput or an MTP speedup. No llama.cpp compatibility or long-context result is claimed.
| Artifact | Tensors | Shards | Size |
|---|---|---|---|
| Q8_0 reference, protected BF16 ingress/attention | 888 | 20 | 262.704 GiB |
| MTP BF16 sidecar, all 8 dense draft layers | 160 | 1 | 4.183 GiB |
The Q8_0 reference uses Q8_0 for embedding/head, dense MLP matrices, shared experts and all routed experts. Attention, vision/audio, norms, convolutions, router weights/bias/scales preserve their original BF16/F32 bytes. It is a Q8 matrix reference with native anchors, not a uniformly Q8 model. It was quantized directly from the original BF16 checkpoint, without imatrix. MTP is kept separately and is not required for ordinary decoding.
general.architecture = inkling, inkling.tensor_layout = source-interleaved-v1.
Every tensor keeps its original checkpoint name, shape and row order.
w13 rows are [gate0, up0, gate1, up1, ...]; experts are never removed,
merged or reordered. All 42 decoder layers and 256 experts per sparse layer
are present. Full tokenizer/config/processor JSON and chat template are embedded
in the first shard and also supplied as sidecars. This is a custom explicit
contract, not an existing llama.cpp tensor-name convention.
Each artifact directory contains tensor-plan.json, audit.json and
SHA256SUMS. Audits check all tensor names/shapes/types/byte counts, native
payload bit equality, sampled Q8 dequantization error and every file hash.
These structural checks do not establish model-output quality or runtime support.
hf download Baekpica/Inkling-Small-GGUF --include 'Q8_0/*' --local-dir ./inkling
cd inkling/Q8_0
sha256sum -c SHA256SUMS
Source: Thinking Machines Inkling-Small, Apache-2.0.
GGML quantizers: Baekpica/ds4-dfm-rs,
pinned 0bf398f459987c611293aa889772c80e272f045d.
Support my work
I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.
While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.
Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.