Ling-3.0-Tiny APEQ SubMini (GGUF)
This repository contains one custom GGUF quantization of inclusionAI/Ling-3.0-tiny, made for local llama.cpp inference where storage and decode speed matter.
It is an APEQ-style, per-tensor mixed-precision profile. It is not a standard uniform Q5_K quantization: the sparse MoE FFN uses Q2_K, most unchanged tensors use Q6_K/Q8_0/F32, and selected later attention tensors use Q5_K.
File
| File | Size | SHA-256 |
|---|---|---|
Ling-3.0-Tiny-APEQ-SubMini-3.06GB.gguf | 3,057,430,432 bytes (2.85 GiB) | 5565bad712e9915ba917485a0030d1c4d81765008be27a5b716acb7b4c3e04c1 |
The file is 6,897,472 bytes (0.225%) smaller than the local Mini comparator used during this experiment.
Quick start — llama.cpp
Use a recent llama.cpp build with support for the Ling / bailingmoe3 architecture.
llama-cli \
-m Ling-3.0-Tiny-APEQ-SubMini-3.06GB.gguf \
-ngl 99 \
-cnv \
--reasoning off
To start an OpenAI-compatible local server:
llama-server \
-m Ling-3.0-Tiny-APEQ-SubMini-3.06GB.gguf \
-ngl 99 \
-sm none \
-c 4096 \
--reasoning off
The base model's native chat template is embedded in the GGUF. The original model recommends temperature=1.0, top_p=0.95, and top_k=20; use deterministic settings only when reproducing the comparison below.
Validation
This is a local comparative result, not a claim of universal quality equivalence.
The final artifact was regenerated twice from BF16 with the same exact tensor map and had the same SHA-256 each time. On a fresh, disjoint 24-case deterministic ZxBench-derived holdout, it matched the Mini comparator exactly:
| Measure | Mini comparator | This GGUF |
|---|---|---|
| Holdout mean score | 73.292 | 73.292 |
| Cases scoring >=80 | 13/24 | 13/24 |
| Nonterminal/empty results | 3 | 3 |
| V100 decode median | 120.658 tok/s | 131.810 tok/s |
| V100 fixed-512 prefill | 1,846.080 tok/s | 1,723.908 tok/s |
Throughput was measured with llama.cpp build b10516 on a Tesla V100-SXM2-16GB, full GPU offload, no speculative decoding, and three samples of a 512-token prefill / 128-token decode workload. Hardware, context length, batching, runtime build, and sampling parameters will affect results.
The holdout used deterministic local graders adapted from ZxBench scenarios. It is a representative comparative screen, not a full hosted ZxBench run with optional AI judges or sandboxes.
Quantization profile
| Tensor type | Count |
|---|---|
| F32 | 215 |
| Q2_K | 72 |
| Q5_K | 62 |
| Q6_K | 102 |
| Q8_0 | 75 |
The profile preserves higher precision in the early sensitive attention path and lowers standard attention tensors in blocks 5–23, plus blk.4.attn_q.weight, to Q5_K. Broader reductions were screened and rejected due to measurable regressions.
License and attribution
This quantization is derived from inclusionAI/Ling-3.0-tiny and follows its MIT license. Please review the upstream model card for the original model's capabilities, limitations, and usage guidance.