Magistral-Small-2509-MXFP4-GGUF

GGUF quantization of mistralai/Magistral-Small-2509 — a 24B multimodal reasoning model fine-tuned from Mistral-Small-3.2-24B-Instruct-2506. Optimized for agentic coding, tool calling, and multi-step reasoning with vision support.

Quantized to MXFP4 format for efficient inference with minimal quality loss.

About MXFP4

MXFP4 (Microscaling FP4, E2M1) is an open standard 4-bit format under the OCP Microscaling Formats specification. It uses E2M1 layout with block-wise shared 8-bit scaling factors, working on any GPU or CPU without hardware-specific acceleration.

Files

Quantization Details

Model Description

• Developer: Mistral AI
• Base Model: Mistral-Small-3.2-24B-Instruct-2506
• Architecture: Dense transformer, Mistral3ForConditionalGeneration
• Parameters: 24B
• Context Length: 131,072 tokens (native)
• Vision: 24-layer Pixtral ViT encoder (1024 hidden)
• Capabilities: Agentic coding, tool calling, multi-step reasoning, image understanding
• Languages: Multilingual (27+ languages)
• License: Apache 2.0

Usage

llama.cpp (CLI)

# Text + Image
llama-cli -m magistral-small-2509-mxfp4.gguf \
  --mmproj mmproj-magistral-small-2509-f16.gguf \
  --image photo.jpg \
  -p "Describe this image in detail" \
  -n 512

# Text only
llama-cli -m magistral-small-2509-mxfp4.gguf \
  -p "Solve this step by step: 23 * 47" \
  -n 512

# OpenAI-compatible server
llama-server -m magistral-small-2509-mxfp4.gguf \
  --mmproj mmproj-magistral-small-2509-f16.gguf \
  --port 8080

llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="FreedomAISVR/Magistral-Small-2509-MXFP4-GGUF",
    filename="magistral-small-2509-mxfp4.gguf",
    n_gpu_layers=-1,
)

response = llm.create_chat_completion([
    {"role": "user", "content": "Write a Python function to sort a list"}
])
print(response["choices"][0]["message"]["content"])

Direct download

from huggingface_hub import hf_hub_download

for filename in ["magistral-small-2509-mxfp4.gguf", "mmproj-magistral-small-2509-f16.gguf"]:
    hf_hub_download(
        repo_id="FreedomAISVR/Magistral-Small-2509-MXFP4-GGUF",
        filename=filename,
        local_dir="./models"
    )

Quantization Pipeline

1. Download source weights
   huggingface_hub.snapshot_download("mistralai/Magistral-Small-2509")

2. Convert text model to F16 GGUF
   convert_hf_to_gguf.py --outtype f16

3. Extract vision encoder
   convert_hf_to_gguf.py --mmproj --outtype f16

4. Quantize to MXFP4
   llama-quantize magistral-small-2509-f16.gguf magistral-small-2509-mxfp4.gguf MXFP4

Note: This model requires mistral-common to be installed for tokenizer conversion:

pip install mistral-common[image,audio]

Hardware

License

Apache 2.0 — same as the original mistralai/Magistral-Small-2509.