Qwen3.8-Flash-Next MLX Mixed 2-bit/4-bit
This is a reproducible MLX conversion of
Qwen/Qwen3.8-Flash-Next at immutable
revision f5d08274bafd880402bd16f5e3e6c514136ec06c.
The 48 base routed-expert banks use MLX affine Q2/group-128, the 160-wide n-gram table and remaining eligible core/MTP matrices use Q4, while token/output embeddings, QSA indexers, routers, vision, and MTP fusion heads remain BF16. This is the 128 GB Mac profile.
- Artifact payload: 68.08 GiB
- Quantized Q2 modules: 144
- Quantized Q4/group-32 modules: 128
- Quantized Q4/group-64 modules: 682
- Source: 180B parameters including 125B main, 51B n-gram embedding, and 4B MTP
- Native model context: 262,144 tokens; see the v0.45.0 runtime limit below.
Run locally with mere.run
Native support is available as of mere.run v0.45.0. The instructions below use its Apple Silicon Swift/MLX runtime. No Python environment, local quantization, or separate MTP-model download is needed.
Requirements and current limits
- Apple Silicon Mac, macOS 15 or later, with 128 GB unified memory recommended.
- Enough free disk space for the approximately 68.08 GiB checkpoint; a fresh v0.45.0 pull budgets about 82 GiB including safety headroom. Leave memory available for macOS and runtime buffers.
- In v0.45.0, text generation is qualified; image input is not yet qualified,
despite the model's upstream multimodal architecture and
vision-chatID. - Prompt plus requested output must fit within 2,048 tokens, including chat
template/system tokens. The upstream 262,144-token context is not yet available
in this runtime: long-context QSA selection is still being integrated. Requests
above the current budget are rejected. Set
--max-tokensexplicitly; the generic 2,048-token output default leaves no room for a prompt.
1. Install or update mere.run
Download mere.run v0.45.0 or newer from the mere.run releases page, install MereRun in Applications, then open the app's Settings and choose Install CLI. Existing installations can use MereRun > Check for Updates… and reinstall the CLI from Settings if needed.
mere.run --version
This should report 0.45.0 or later. See the
getting-started guide for installation details.
2. Pull this pre-quantized checkpoint
After reviewing the Qwen Community License 1.0:
mere.run model pull vision-chat-q38-flash-next-mixed \
--accept-license-terms
Use this managed model ID, not the Hugging Face repository name, with the commands below. It resolves to this Sawfwair mixed Q2/Q4 distribution and includes the bundled MTP head. The repository is public and ungated; the acceptance flag records acknowledgement of the model's terms, not a request for gated access.
For an external SSD, add --cache-dir /Volumes/Models/huggingface-cache to the
pull command, substituting your mounted drive's path. Keep that drive mounted
while using the model. To inspect support and download requirements without
downloading, add --preflight --json to the pull command.
3. Generate text with verified MTP
mere.run text chat \
--model vision-chat-q38-flash-next-mixed \
--context-size 2048 \
--max-tokens 256 \
--temperature 0 \
--no-thinking \
--stream \
--stats \
--prompt "Explain sparse attention in three short sentences."
--temperature 0 selects greedy decoding, which uses the bundled MTP draft head
by default for eligible requests. Proposed tokens are verified by the full target
model. No opt-in flag or extra checkpoint is required. --stats reports local
generation timing; --no-thinking keeps this short example focused on the answer.
For a target-only comparison or to avoid loading the draft head, prefix the same
command with MERERUN_Q35_MTP_SPECULATION=0.
4. Serve a local OpenAI-compatible API
After the pull completes, start the server:
mere.run api serve \
--engine text-chat-q36 \
--model vision-chat-q38-flash-next-mixed \
--context-size 2048 \
--host 127.0.0.1 \
--port 8080
text-chat-q36 is the shared Qwen-family serving engine; it also dispatches this
Qwen3.8-Flash-Next model. Startup loads and warms the target and MTP graphs before
the server begins listening, so allow the initial warmup to finish.
From another terminal:
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "vision-chat-q38-flash-next-mixed",
"messages": [{"role": "user", "content": "Explain sparse attention in three short sentences."}],
"max_tokens": 256,
"temperature": 0
}'
The example binds to localhost only. If you have configured MERERUN_API_KEY,
include its bearer token in requests. See the
API guide for authentication and client setup.
Runtime status
The tensor inventory, source hashes, MLX packing, fused-expert split, convolution
layout, and zero-centered RMSNorm conversion are validated by the bundled
MERERUN_CONVERSION.json. A Qwen4Exp-aware MLX runtime is required; do not
expect an older mlx-lm or mlx-vlm release to dispatch this new architecture.
License
This redistribution retains the upstream Qwen Community License 1.0 in
LICENSE. Review it before use. In particular, it contains attribution/display
requirements for very large commercial products and separate-license conditions
for certain commercial Model-as-a-Service and AI Work Assistant uses. The model
is not gated; downloading or using it does not remove those terms.
The upstream model card is preserved as README.upstream.md.