Escha Qwen3.6-35B-A3B W2 GGUF
I ported EschaLabs/Qwen3.6-35B-A3B-Escha-W2 to llama.cpp. Weights are theirs. This is not a requant.
I decode their native 2-bit eschamoe code in-kernel (GGML_OP_ESCHA_MOE). Expert payloads are the same bytes as the safetensors. Non-expert tensors are fp16.
Stock llama.cpp cannot load this file. You need my fork, branch escha-w2:
https://github.com/Ajay9o9/llama.cpp-escha/tree/escha-w2
If you only download the GGUF, it will not run.
File
| Source checkpoint | This GGUF | |
|---|---|---|
| Size | 12.30 GB | 14.50 GB (+17.9%) |
| Expert code | native 2-bit / 3-bit | same bytes |
| Non-expert tensors | int8 | fp16 |
| Layers | 40 + MTP | 40, MTP dropped |
| Tensors | 976 | |
| VRAM, 32k context | 14.5 GB of 24 |
File: Escha-Qwen3.6-35B-W2.gguf. 256 experts, top-8. gate/up K=2, down K=3. Extra size is fp16 on the non-expert tensors, not the expert code.
Quality vs Escha SGLang
Same GPU. I sent token ids, not text. Tokenizers agreed on 196,608 tokens of wikitext. Perplexity uses the same chunking on both sides (n_ctx 2048, 96 chunks, 98,208 scored tokens).
| Perplexity, wikitext-2 | |
|---|---|
| Escha SGLang | 6.0988 |
| This GGUF | 6.0997 +/- 0.047 |
| Difference | +0.0009 (+0.015%) |
Gap is inside the llama.cpp run's own error bar, and smaller than SGLang's own rerun drift (+0.0016). One corpus. Not a task-accuracy claim.
| Teacher-forced, 97 positions | |
|---|---|
| Top-1 token agreement | 93 / 97 = 95.9% |
| JS divergence over top-20 | mean 1.7e-4, max 1.0e-3 |
| |delta logprob| where they agree | mean 0.013, max 0.072 |
| True-token NLL | SGLang 1.39704, this GGUF 1.38663 |
Same prefix at every position, so a miss at i-1 does not poison i. Two SGLang runs against each other were 96 / 97.
| Greedy, 16 prompts, 64 tokens, temp 0 | |
|---|---|
| Token-identical | 13 / 16 |
| Diverged | 3 / 16, at tokens 5, 26, 49 |
Two SGLang runs were also 13 / 16 identical. Greedy is a hard argmax. I would not hang a quality claim on this.
Op test vs a numpy dense-fold reference (CPU and CUDA scored separately): rel RMS 1.06e-6 at K=2, 9.26e-7 at K=3.
Speed vs Escha SGLang
Single stream, batch 1, one RTX 3090. Their runtime is still ~2.6x faster at generation. Context lengths were not matched, so the ratio is rough.
| Escha SGLang | This GGUF | |
|---|---|---|
| Generation (tok/s) | 110.4 | 42.6 |
| Prefill, 4,566 tokens (tok/s) | 2595 | 634 |
SGLang generation was 256 new tokens with ignore_eos. MTP is dropped here, so speculative decoding on their server is not in these numbers.
Build the fork
git clone -b escha-w2 https://github.com/Ajay9o9/llama.cpp-escha.git
cd llama.cpp-escha
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build -j 12
86 is an RTX 3090. Change that for your GPU. -j 12 is a 12-core CPU.
Download the GGUF
hf download aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF Escha-Qwen3.6-35B-W2.gguf
Direct file: https://huggingface.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF/resolve/main/Escha-Qwen3.6-35B-W2.gguf
Run it
./build/bin/llama-server \
-m Escha-Qwen3.6-35B-W2.gguf \
-ngl 99 --host 127.0.0.1 --port 8080 \
-c 32768 -np 1 -t 12 --jinja
Or:
./build/bin/llama-cli \
-m Escha-Qwen3.6-35B-W2.gguf \
-ngl 99 --jinja -c 32768
It is a reasoning model. Give it room to think or you get an empty answer.
What this is not
- Not a Q4_K / Q8_0 requant of a dense reconstruction
- Not upstream llama.cpp
- Not bit-identical to Escha SGLang on every greedy prompt
- No MMLU / GSM8K / long-context eval
License
Apache-2.0, same as the Escha weights and the escha 1.0.2+qwen3moe wheel I used.
Source: EschaLabs/Qwen3.6-35B-A3B-Escha-W2
Runtime I compared against: escha 1.0.2+qwen3moe on SGLang