Muse-Glimmer-30B - GGUF (AK line)

👋 I built this line solo - the calibration, the per-tensor allocations, and the eval harness behind every number below. I'm looking for internships in AI agent orchestration and model inference. If this work looks relevant to your team: linkedin.com/in/theaaryankapoor

State-of-the-art GGUF quantizations for meta-models/Muse-Glimmer-30B. Eight builds (27.86 B params, 52 dense layers, GQA 32:2), each with a custom per-tensor bit allocation derived for its size point - plus the stock BF16 vision encoder.

Benchmarked head-to-head against the Unsloth, Meta and bartowski lines, every file scored on the same rig against the same BF16 reference: 26 wins, 6 statistical ties, 0 losses across 32 paired comparisons on two evaluation sets.

One line per publisher. Log y, bits-per-weight on the secondary axis, and the crowded 16 GB class magnified.

Every comparison with its 95 % interval - blue clears zero, grey is a statistical tie, and the right column carries the held-out C4 verdict. The full numbers are in the table below.

Comparison set: the three widest-distribution GGUF lines for this model, as published 2026-08-12; the Method section has everything needed to reproduce any number here.

File naming. Every quant in this line carries the AK- prefix: these are custom per-tensor allocations, not llama.cpp's stock recipes, so AK-Q4_K_M and a stock Q4_K_M are different files. mmproj keeps its upstream name.

Which file do I want?

AK-Q4_K_XL is the strongest file in the crowded 16 GB class - no published quant of this model at any comparable size comes within 13 % of it, including Meta's own official quant, which it beats by 13 % while being half a GB smaller. The same story repeats at Q5: AK-Q5_K_M beats Meta's official 19.65 GB kquant-dynamic by 9-13 % on both evaluation sets while being 460 MB smaller. At Q8, AK-Q8_K_L beats Unsloth's build while being smaller.

llama-server -m Muse-Glimmer-30B-AK-Q4_K_XL.gguf \
             --mmproj mmproj-Muse-Glimmer-30B-BF16.gguf -c 8192 -ngl 99

Full measurement table

Intervals are a paired per-token cluster bootstrap over the 60 evaluation chunks. Δ is against the closest-sized non-AaryanK file.

Reading the numbers

Two Q8 builds, two jobs. AK-Q8_K_L is the size-class winner - smaller than Unsloth's Q8 and −20.8 % KLD, confirmed on every slice tested. AK-Q8_K_XL is the maximum-fidelity build: −29.7 % at +8.2 % bytes (10.04 bpw vs 9.28), for when the last 2.7 GB of VRAM is cheaper than the last drop of divergence.

PPL ratio and KLD can rank differently at Q3-Q4. PPL scores only the probability of the true next token; KLD scores the whole distribution - so AK-Q3_K_XL and AK-Q4_K_M lead their size peers on KLD while trailing by ~0.001 on PPL ratio. Both metrics are in the table; the interval column carries the verdicts.

Does the margin generalise?

The same files re-measured on six evaluation sets, four held out and audited at zero fragment overlap with any calibration corpus. 11 of 16 held-out margins exceed the same file's wikitext margin, and every interval in the chart excludes zero - including all four held-out domains for AK-Q8_K_L, where the size-matched Q8 lead spans −17 % to −24 %.

Tail behaviour

Method

• Reference: our own BF16 GGUF, converted with llama.cpp pinned at 62bf73d2. The conversion was checked against every competitor's file across 21 load-bearing KVs, so the comparison measures quantization rather than a conversion delta.
• Eval: llama-perplexity --kl-divergence, ctx 4096 × 60 chunks → 122,820 scored tokens. ctx 4096 matters for this architecture - it alternates 3× sliding-window (2048) with 1× full-attention NoPE layers, and only at ctx ≥ 4096 does every scored token sit beyond the window.
• Statistics: paired per-token cluster bootstrap at the 2047-token chunk width for every interval.
• Confirmation: the Q4 result was re-run under 9 independent calibration draws across three corpus families on an untouched slice - beneficial in 9/9, no reversals, every interval excluding zero, against an MDE fixed before any data was collected.
• Long context: re-measured at ctx 8192; the lead holds and slightly grows.
• Capability: a 130-case tool-calling suite scored as paired agreement with BF16 - AK-Q4_K_M matches BF16 on 128 of 130 cases with one flip in each direction: statistically indistinguishable (exact McNemar p = 1.000).
• Ten pre-registered apparatus gates, all passing, including full-vocabulary agreement with HF transformers and exact greedy generation agreement (235/235 tokens).
• Scope: the text tower is what is measured and quantized; mmproj ships as the stock BF16 encoder. KLD values are model-local (this head applies logit soft-capping) - compare within this table only.

Held-out sets: GitHub source (numpy/redis/django/sqlite/nlohmann), OASST dialogue, GSM8K + arXiv abstracts, and Wikipedia in sixteen non-English languages.

Per-tensor bit allocation derived separately at each target bit-width using importance data from a diverse in-house calibration set. Base model licence and usage policy unchanged from meta-models/Muse-Glimmer-30B (Apache-2.0).