Qwen3.8-27B-ANE
INT4 runtime package for Qwen3.8-27B on Apple Silicon, for use with qwen-ane.
Learned matmuls, Gated DeltaNet recurrence, attention, and MLP tails run on the Apple Neural Engine through AppleNeuralEngine.framework. The Metal GPU is not used. This is not the Flash-Next hybrid checkpoint (ANE attention + GPU MoE), and it is not a Transformers-loadable full-weight dump.
The Hub safetensors widget reports only the host tensors in model.safetensors (~2.5B F16 values). The 27B INT4 layer weights are in quant_cache/.
Package layout
| Path | Size | Role |
|---|---|---|
model.safetensors | 5.09 GB | Host embeddings, RMSNorms, and LM head (F16) |
quant_cache/ | 9.98 GB | Pre-quantized INT4 blobs, per-row F16 scales, and manifest.json |
mtp.safetensors | 0.85 GB | Optional MTP draft-layer tensors (BF16) |
| tokenizer / config | — | config.json, tokenizer.json, chat_template.jinja, and related files |
Total package is about 16 GB. First load uses the packaged INT4 cache; it does not re-quantize from BF16.
Requirements
- Apple Silicon (developed and measured on an M5 Max, macOS 27)
- qwen-ane
AppleNeuralEngine.framework(present on macOS; the runtime talks to it through the vendored driver)
Install and run
git clone https://github.com/True2456/qwen-ane.git
cd qwen-ane
pip install -e .
Download into ~/.qwenANE/models/27b:
qwen-ane pull 27b
Or:
hf download True2456/Qwen3.8-27B-ANE --local-dir ~/.qwenANE/models/27b
Chat and serve:
qwen-ane chat -model 27b
qwen-ane serve -model 27b -port 2457
Optional speculative decode (--mtp-draft 2 uses mtp.safetensors):
qwen-ane serve -model 27b -port 2457 --mtp-draft 2
Coding agents can point an OpenAI-compatible client at that server. A Pi provider lives in the qwen-ane tree as extensions/pure27-pi.ts.
Measured performance (M5 Max)
Figures below are from qwen-ane on an Apple M5 Max running 100% on-chip Apple Neural Engine inference (0% Metal GPU).
Context scale & power metrics (2026-09-25)
Cold prefix (reset between lengths, salted prompts, reused=0). Pure ANE server context 8576, INT4 weights. Watts are mean powermetrics --samplers cpu_power,gpu_power,ane_power -i 500 over the generate. PeakMem is footprint -p phys_footprint_peak. Prompt tokens were 1000 / 1999 / 3997 / 7993; completion 128.
| Test | TTFT(ms) | TPOT(ms) | ppTPS | tgTPS | E2E(s) | Throughput | PeakMem | ANE / GPU / CPU W |
|---|---|---|---|---|---|---|---|---|
| pp 1024 / tg 128 | 31856.7 | 251.4 | 31.4 | 4.0 | 63.8 | 17.7 | 21.0 GB | 2.86 / 0.24 / 7.04 |
| pp 2048 / tg 128 | 69915.0 | 260.6 | 28.6 | 3.8 | 103.0 | 20.6 | 21.0 GB | 2.67 / 0.11 / 5.91 |
| pp 4096 / tg 128 | 165730.8 | 278.2 | 24.1 | 3.6 | 201.1 | 20.5 | 21.0 GB | 2.44 / 0.17 / 6.27 |
| pp 8192 / tg 128 | 439329.1 | 319.8 | 18.2 | 3.1 | 480.0 | 16.9 | 21.0 GB | 2.23 / 0.25 / 7.34 |
- Zero GPU utilization: Metal GPU stays idle at 0.1–0.2 W while the 27B model runs entirely on the ANE.
- Ultra-low power draw: Sustained ANE power draw is only ~2.2–2.9 W; total package power stays under 10 W.
- Rock-solid memory footprint: Physical memory footprint stays flat at 21.0 GB from 1k through 8k+ tokens.
Microbenchmarks & Agent turns
| Workload | Result |
|---|---|
| Greedy decode, short / agent turns | about 3.4–4.3 tok/s |
| MTP-2, repetitive capitals probe | 5.58 tok/s vs 4.34 greedy; unique prose often falls back toward greedy |
| Prefill, 300-token agent prompt | 5.27 s TTFT (~57 tok/s), width-64 / 32-lane programs |
| Prefill, 4080-token cold prompt | 26.71 tok/s at the earlier 16-lane chunked-GDN config |
| Decode power (documented ANE-only path) | about 6 W; GPU idle |
Prefix cache is on for the HTTP server: a follow-up agent turn that reuses 300 prompt tokens prefills the 56-token suffix in about 1.2 s on the same machine.
Decode is slower than GPU INT4 (MLX) because each token walks 64 layers as many small ANE programs rather than one large GPU GEMM. INT4 shrinks the package and the DRAM traffic; the ANE datapath still executes in F16.
Architecture
64 layers in a 3:1 pattern: 48 Gated DeltaNet (linear attention) layers and 16 full-attention layers (every fourth layer). Projections, convolutions, and SwiGLU MLPs are compiled MIL graphs on the Neural Engine. Host NumPy packs IO, samples tokens, and holds GDN / KV state.
License
Apache-2.0. Base weights: Qwen Team. INT4 packaging and ANE runtime: True2456/qwen-ane.