Qwen3.8 Flash-Next MLX SSD Stream
Quality-first Flash-Next at 70+ tok/s on a 128 GB Mac
This prepared model prioritizes quality and speed: a BF16 core and verification path, Q4 routed experts, native MTP, multimodal support, and the full 262,144-token context window. On an M4 Max with 128 GB unified memory, matched warm workloads measured 71-74 tok/s for code and technical prose.
SSD Stream makes that configuration fit without keeping the model's 51.2 GB lookup table in unified memory. The table streams from SSD while the GPU is already working, leaving that memory available for model weights, KV cache, and macOS. A Q6 copy of the vocabulary head proposes MTP tokens that are always checked by the untouched BF16 verification head.
What this model is
| Component | Source |
|---|---|
| Transformer and vision weights | Qwen/Qwen3.8-Flash-Next at de4b8e4d43b917e7706784d8bb445c9af86a3540 |
| FP8 lookup table | RadixArk/Qwen3.8-Flash-Next-NVFP4 at 7b719225242aacd3dbd3f9407468c2ee9a9d2594 |
| Serving engine | garnermccloud/mlx-serve |
| Tested hardware | Apple M4 Max with 128 GB unified memory |
| Prepared download | Approximately 127 GB |
The prepared repository contains the complete model and a read-only
ngram_table.bin; no conversion step or companion draft model is required.
The table payload is source-verified during conversion and remains in its
original FP8 E4M3 format.
Performance
M4 Max, 128 GB
| Result | Value |
|---|---|
| Native context | 262,144 tokens |
| Model allocation before request caches | Approximately 75.3 GB |
| SSD-streamed lookup table | 51.2 GB |
| Complete prepared repository | Approximately 127 GB |
| Long-context prefill | 280.7 prompt tok/s at 248,445 tokens |
The decode comparison used the 262,144-token server context and native MTP at depth 4. Rates are completion tokens divided by total request wall time, median of three warm runs after one discarded warmup. Each pair used the same prompt and greedy sampling.
| Workload | Completion | Native MTP | Serial | Speedup |
|---|---|---|---|---|
| Python CSV parser module | 600 tokens | 73.8 tok/s | 33.2 tok/s | 2.22x |
| Technical Rayleigh-fading prose | 600 tokens | 71.2 tok/s | 33.0 tok/s | 2.16x |
| Numbered unique sentences | 1,024 tokens | 54.4 tok/s | 32.2 tok/s | 1.69x |
The uncached 248,445-token retrieval test returned its exact buried key and stopped normally within the native context window. The release also passed 39 end-to-end checks covering text, structured tools, streaming, MTP, long-context retrieval, images, concurrent decode, and mixed speculative and serial traffic.
Run it
Build mlx-serve, then give it this Hugging Face model ID:
git clone --recurse-submodules https://github.com/garnermccloud/mlx-serve
cd mlx-serve
brew bundle install --file=Brewfile
./app/build.sh
./zig-out/bin/mlx-serve run garnermccloud/Qwen3.8-Flash-Next-MLX-SSD-Stream
The first run downloads the prepared files and then starts the server at
http://localhost:11234. Later launches reuse the verified local snapshot.
Native MTP and SSD streaming are selected from the model metadata, so there are
no model-specific serving flags.
Capabilities
- 262,144-token native context
- text, image, and video input
- reasoning and structured tool calls
- OpenAI Chat Completions, Responses, Anthropic Messages, and Ollama APIs
- native MTP enabled automatically; no companion model or serving flags
Precision
| tensors | width |
|---|---|
| routed experts, including MTP | Q4 affine, group 64 |
| draft-only vocabulary head | Q6 affine, group 64 |
| core, embeddings, verification head, vision | BF16 |
| n-gram table | source FP8 E4M3, streamed from SSD |
How SSD Stream works
Flash-Next needs 16 deterministic lookup rows per token. Their addresses are
known before the model reaches the block that consumes them, so mlx-serve
starts persistent parallel pread operations while the preceding GPU work is
still running. It restores rows in token order, converts FP8 directly to BF16,
and waits only when storage takes longer than the overlapping computation.
The 51.2 GB table is mapped read-only. SSD Stream does not create a second private copy or reserve a multi-gigabyte cache; macOS may retain useful pages in its reclaimable filesystem cache.
Hugging Face metadata
Hugging Face's automatic parameter and precision labels describe the packed
safetensors, not the complete source architecture. The transformer has 125B
parameters with 6B active per token. The architecture also includes the
51.2-billion-element PLE table and MTP weights, bringing its logical total near
180B. The PLE is stored separately in ngram_table.bin, so it is not included
in the safetensors parameter count.
License and attribution
The model retains the Qwen Community License. Transformer and vision weights come from Qwen. The FP8 table comes from the RadixArk NVFP4 checkpoint. Serving support and the prepared layout are provided by garnermccloud/mlx-serve, an independent fork of mlx-serve.