Qwen3.8-9B MLX MXFP4

This repository contains an MLX MXFP4 quantization of empero-ai/Qwen3.8-9B, prepared for local inference on Apple silicon.

Qwen3.8-9B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-9B architecture. The source model was trained by Empero on approximately 70,000 curated teacher traces covering mathematics, code, general reasoning, instruction following, and tool use.

Model details

The model uses weight-only MXFP4 quantization. Non-quantized tensors retain the dtype selected from the source configuration (bfloat16). The tokenizer, generation configuration, and chat template are included in this repository.

Requirements

• An Apple silicon Mac
• A recent version of macOS
• Python 3.10 or newer (Python 3.12 tested)
• mlx-lm 0.31.3 or newer
• mlx 0.32.0 or newer

Install the runtime:

python -m pip install --upgrade mlx-lm

Quick start

Hugging Face model ID: schsu/Qwen3.8-9B-MLX-MXFP4.

Command line

mlx_lm.generate \
  --model schsu/Qwen3.8-9B-MLX-MXFP4 \
  --prompt "Explain why the sky is blue." \
  --max-tokens 2048 \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20

The model normally emits a reasoning block before its final answer. To disable the explicit thinking block for short responses:

mlx_lm.generate \
  --model schsu/Qwen3.8-9B-MLX-MXFP4 \
  --prompt "Reply with exactly: Hello" \
  --max-tokens 64 \
  --chat-template-config '{"enable_thinking": false}'

Python

from mlx_lm import generate, load
from mlx_lm.sample_utils import make_sampler

model_id = "schsu/Qwen3.8-9B-MLX-MXFP4"
model, tokenizer = load(model_id)

messages = [
    {"role": "user", "content": "Explain why the sky is blue."},
]
prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
    enable_thinking=True,
)

response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=2048,
    sampler=make_sampler(temp=0.6, top_p=0.95, top_k=20),
)
print(response)

OpenAI-compatible server

mlx_lm.server \
  --model schsu/Qwen3.8-9B-MLX-MXFP4 \
  --host 127.0.0.1 \
  --port 8080

Example request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "schsu/Qwen3.8-9B-MLX-MXFP4",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.6,
    "top_p": 0.95,
    "max_tokens": 512
  }'

Function calling

The bundled Qwen chat template supports function definitions and multi-turn tool results. Applications should preserve the complete conversation and add each tool result with role: "tool" before requesting the next assistant turn.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

messages = [{
    "role": "user",
    "content": "What is the weather in Taipei? Use the tool.",
}]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    add_generation_prompt=True,
    tokenize=False,
    enable_thinking=False,
)

The model's native template renders tool calls using <tool_call> and <function=...> XML-style tags. Tool parsers must support this Qwen3.5 format. When implementing an agent loop, do not treat the end of a tool call as the end of the entire task.

Recommended generation settings

The source model recommends:

Allow enough output tokens for reasoning tasks. Very short generation limits may stop inside the model's thinking block, while greedy decoding over long outputs can increase the chance of repetition. For interactive applications, start with 2,048 to 8,192 output tokens and adjust for the task and available memory.

The architecture supports a context length of up to 262,144 tokens, but the practical limit depends on unified memory because the KV cache grows with the active context.

Local conversion

This checkpoint was produced from the source BF16 weights with:

mlx_lm.convert \
  --hf-path empero-ai/Qwen3.8-9B \
  --mlx-path Qwen3.8-9B-MLX-MXFP4 \
  --quantize \
  --q-mode mxfp4 \
  --q-group-size 32 \
  --q-bits 4

Conversion environment:

• mlx-lm==0.31.3
• mlx==0.32.0
• transformers==5.15.0
• Python 3.12

Smoke-test results

The converted checkpoint was loaded and tested locally on an Apple silicon Mac with 24 GB of unified memory.

These numbers are a functional smoke test, not a comprehensive benchmark. Throughput varies with Mac model, prompt length, thermals, and generation settings.

Evaluation

No full downstream benchmark suite has been run specifically on this MXFP4 checkpoint. The following results are reported by the source model authors for the unquantized Qwen3.8-9B checkpoint and should not be interpreted as measured MXFP4 results.

See the source model card for the authors' evaluation methodology and additional details.

Limitations

• Quantization can change output probabilities and may reduce accuracy relative to the BF16 source model.
• The model can produce incorrect, biased, or fabricated information. Verify important outputs independently.
• Function-calling reliability depends on the surrounding tool parser, chat template handling, stopping criteria, context management, and agent-loop limits.
• The fine-tuning data was text-only. Vision behavior is inherited from the Qwen3.5 base and was not evaluated by the source model authors or during this conversion.
• Long-context behavior was not validated at the full architectural limit.
• This model is not intended to provide professional medical, legal, financial, or safety-critical advice without qualified human review.

License and attribution

The weights are distributed under the Apache License 2.0, following the source model and its Qwen3.5-9B base. Users are responsible for reviewing and complying with the applicable license and policies.

• Source fine-tune: empero-ai/Qwen3.8-9B
• Base model: Qwen/Qwen3.5-9B
• Source model developer: Empero
• MLX framework: ml-explore/mlx
• MLX language-model tooling: ml-explore/mlx-lm

Acknowledgements

Thanks to Empero for releasing Qwen3.8-9B, the Qwen team for Qwen3.5-9B, and Apple's MLX team for the Apple-silicon inference and quantization ecosystem.