Qwen3.6-35B-A3B-NSC-ACE-SABER GGUF MTP
This repository hosts the MTP-oriented llama.cpp/GGUF builds for
GestaltLabs/Qwen3.6-35B-A3B-NSC-ACE-SABER. The source checkpoint is the full
safetensors model in GestaltLabs/Qwen3.6-35B-A3B-NSC-ACE-SABER.
These files are rebuilt separately from the first GGUF release using a fresh MTP-aware conversion path. Use these artifacts when your runtime supports Qwen MTP / multi-token prediction acceleration.
Image/video sidecars: This repository now includes the restored Qwen3.6 multimodal config, processor/preprocessor files, tokenizer/chat template, safetensors index, and
model-vision-from-qwen3.6-base.safetensorsvisual tower sidecar. The existing.ggufbinaries were not rewritten in this metadata-copy pass.
Current Status
Files are published only after the rebuilt F16 GGUF verifies as MTP/NextN-capable
from its actual metadata/tensor layout. Qwen HF tensors named mtp.* are remapped
by the MTP-aware llama.cpp converter into GGUF blk.*.nextn.* tensors plus the
nextn_predict_layers metadata key. The upload worker refuses to publish quants
until the verified F16 marker exists.
Release Snapshot
| Item | Value |
|---|---|
| Source checkpoint | GestaltLabs/Qwen3.6-35B-A3B-NSC-ACE-SABER |
| Base model | Qwen/Qwen3.6-35B-A3B |
| Format | GGUF for llama.cpp-compatible runtimes |
| Conversion target | MTP-aware GGUF export from the llama.cpp MTP branch |
| Quantization range | F16, Q8_0, Q6_K, Q5_K_M, Q5_K_S, Q4_K_M, Q4_K_S, Q3_K_L, Q3_K_M, Q3_K_S, Q2_K |
| Final source compliance | 98.33% on HarmBench-300 |
| Final source KLD | 0.025383937664711 |
| BFCL average plotted improvement | +2.87 percentage points |
Benchmark Plots
BFCL Tool-Calling Check
The source safetensors checkpoint was compared against Qwen/Qwen3.6-35B-A3B
on a 40-case BFCL subset: 20 simple and 20 multiple-function prompts. GGUF files
inherit from that checkpoint, but individual quants should be rechecked if exact
tool-call behavior matters.
| Metric | Base | NSC-ACE SABER source |
|---|---|---|
| Tool-call rate | 92.50% | 95.00% |
| Function name accuracy | 92.50% | 95.00% |
| Required argument name accuracy | 90.00% | 93.12% |
| Required argument value accuracy | 79.79% | 83.54% |
| Exact required-call accuracy | 75.00% | 77.50% |
Available Files
| File | Status | Notes |
|---|---|---|
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-F16.gguf | uploaded | Full GGUF conversion source / highest local fidelity |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q8_0.gguf | uploaded | Near-full quality, large local file |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q6_K.gguf | uploaded | High-quality local default if memory allows |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q5_K_M.gguf | uploaded | Strong quality/size balance |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q5_K_S.gguf | uploaded | Smaller Q5 option |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q4_K_M.gguf | uploaded | Common balanced local target |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q4_K_S.gguf | uploaded | Smaller Q4 option |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q3_K_L.gguf | uploaded | Lower-memory Q3 option |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q3_K_M.gguf | uploaded | Smaller Q3 balance |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q3_K_S.gguf | uploaded | Small Q3 option |
Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q2_K.gguf | uploaded | Minimum-size target; quality loss expected |
The uploader refreshes this card as each artifact finishes. Uploaded non-F16 files are deleted from the build pod after upload to stay under the pod volume quota.
Which Quant Should I Use?
| Quant | Best fit |
|---|---|
| F16 | Maximum fidelity when disk/RAM are not a concern |
| Q8_0 | Very high fidelity local inference |
| Q6_K | Recommended high-quality local starting point |
| Q5_K_M | Strong balance for quality and size |
| Q4_K_M | Practical default for constrained machines |
| Q3_K_M / Q3_K_S | Low-memory experiments |
| Q2_K | Smallest target; use only when memory is the hard constraint |
For agentic/tool-calling workloads, prefer Q6_K, Q5_K_M, or Q4_K_M when possible. Very low quants can affect formatting, argument fidelity, and refusal calibration.
MTP Notes
- These are separate MTP-oriented exports; do not assume the original GGUF repo exposes MTP behavior in runtimes that require MTP metadata/tensors.
- MTP speedups depend on runtime support. Use a current llama.cpp build.
- Quantized body weights keep
blk.*.nextn.*tensors at Q8_0 where supported, because draft-head quality affects speculative acceptance. - The source model's final release metrics are measured before quantization.
- Quantized files should be re-evaluated if exact compliance/KLD behavior matters.
Running With llama.cpp
llama-cli \
-m Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q5_K_M.gguf \
-c 32768 \
-ngl 999 \
-p "Write a compact tool plan for indexing a Python repo."
For OpenAI-compatible local serving:
llama-server \
-m Qwen3.6-35B-A3B-NSC-ACE-SABER-MTP-Q5_K_M.gguf \
-c 32768 \
-ngl 999 \
--jinja
Related Repositories
- Full safetensors checkpoint:
GestaltLabs/Qwen3.6-35B-A3B-NSC-ACE-SABER - Non-MTP GGUF release:
GestaltLabs/Qwen3.6-35B-A3B-NSC-ACE-SABER-GGUF - Base model:
Qwen/Qwen3.6-35B-A3B