gpt-oss-20b — GGUF quantizations
Three llama.cpp GGUF quantizations of openai/gpt-oss-20b, prepared for edge / on-premise deployment with llama-cpp-python and llama.cpp.
Files
| File | Quantization | Approx. size | RAM/VRAM (n_ctx=8k) | Quality |
|---|---|---|---|---|
gpt-oss-20b-Q4_K_M.gguf | Q4_K_M | ~8 GB | ~9 GB | Good — recommended for 12 GB GPUs |
gpt-oss-20b-Q5_K_M.gguf | Q5_K_M | ~10 GB | ~11 GB | Better — recommended for 16 GB GPUs |
gpt-oss-20b-Q8_0.gguf | Q8_0 | ~15 GB | ~16 GB | Best — recommended for 24 GB GPUs |
Sizes are approximate and depend on tokenizer/vocab size after conversion.
Usage
llama-cpp-python (Python)
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="ggfox00000/gpt-oss-20b-GGUF",
filename="gpt-oss-20b-Q4_K_M.gguf", # or Q5_K_M / Q8_0
n_ctx=8192,
n_gpu_layers=-1, # offload everything to GPU (CUDA build required)
)
out = llm.create_chat_completion(
messages=[
{"role": "system", "content": "Tu es un assistant utile."},
{"role": "user", "content": "Résume cette réunion en français."},
],
temperature=0.2,
max_tokens=1024,
)
print(out["choices"][0]["message"]["content"])
For GPU acceleration with NVIDIA CUDA, install the matching wheel:
pip install --no-cache-dir --index-url https://abetlen.github.io/llama-cpp-python/whl/cu121 \
llama-cpp-python
llama.cpp (CLI)
# Pull a specific file
huggingface-cli download ggfox00000/gpt-oss-20b-GGUF gpt-oss-20b-Q4_K_M.gguf \
--local-dir ./gguf
# Run with llama-cli
./llama-cli -m ./gguf/gpt-oss-20b-Q4_K_M.gguf \
-n 512 --temp 0.2 -p "Quel est le meilleur réglage pour la cuisson d'un steak ?"
How these quants were made
huggingface_hub.snapshot_download("openai/gpt-oss-20b")— sharded safetensorsllama.cpp/convert_hf_to_gguf.py … --outtype f16→ F16 GGUFllama.cpp/build/bin/llama-quantize <fp16.gguf> <out.gguf> <quant>for each of Q4_K_M, Q5_K_M, Q8_0
See scripts/quantize_to_gguf/quantize.sh (in the original Gilbert × May-IA project) for the full reproducible workflow.
License
Inherits the Apache 2.0 license of the source model. See the base model card for terms.
Citation
If you use these quantizations, please also cite the original Mistral AI model.
Acknowledgements
- Mistral AI for the base model
- llama.cpp (Georgi Gerganov & contributors) for the quantization tooling