gpt-oss-20B-MXFP4-MOE-GGUF
MXFP4_MOE GGUF quantization of openai/gpt-oss-20b.
About MXFP4_MOE
MXFP4_MOE is the MoE variant of MXFP4 (OCP Microscaling FP4 E2M1 format). It applies a mixed-precision quantization:
| Tensor Group | Quantization | Bits/Param |
|---|---|---|
| Expert FFN weights | MXFP4 (block FP4) | ~4.25 |
| Attention weights | Q8_0 | 8 |
| Router/norms/biases | F32 | 32 |
Total: ~4.63 bits/param — the file is 11.28 GiB for 20.91B total parameters.
MXFP4 runs on any GPU or CPU (unlike NVFP4 which requires Blackwell).
Files
| Filename | Type | Size | Description |
|---|---|---|---|
gpt-oss-20B-MXFP4_MOE.gguf | MXFP4_MOE | 11.28 GiB | Main model weights |
Usage
llama.cpp CLI
llama-cli -hf FreedomAISVR/gpt-oss-20B-MXFP4-MOE-GGUF -cnv -p "You are a helpful assistant"
llama-server
llama-server -hf FreedomAISVR/gpt-oss-20B-MXFP4-MOE-GGUF --ctx-size 0 --jinja
llama-cpp-python
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="FreedomAISVR/gpt-oss-20B-MXFP4-MOE-GGUF",
filename="gpt-oss-20B-MXFP4_MOE.gguf",
)
Quantization Pipeline
# 1. Convert HF model to intermediate GGUF
python convert_hf_to_gguf.py ./models/gpt-oss-20b/ --outfile gpt-oss-20b-f16.gguf --outtype f16
# 2. Quantize to MXFP4_MOE (requires --allow-requantize due to converter auto-type assignment)
llama-quantize --allow-requantize gpt-oss-20b-f16.gguf gpt-oss-20b-mxfp4_moe.gguf MXFP4_MOE
Hardware
| GPU | VRAM | Notes |
|---|---|---|
| NVIDIA RTX 5060 Ti | 16 GB | Quantization performed on this GPU |
License
Apache-2.0 (same as openai/gpt-oss-20b)