KATCoder-2.5-Dev to ROCmFPX GGUF Conversion Guide with MTP Header
This repository contains instructions and scripts to convert the Kwaipilot/KATCoder-2.5-Dev 35B model into the AMD hardware-optimized ROCmFPX GGUF format (such as ROCmFP4) using charlie12345/ROCmFPX and llama.cpp compilation tools.
ROCmFPX allows up to upto avg 80-90 tps text generation speeds on AMD hardware (like Strix Halo / gfx1151, RDNA2, RDNA3, and RDNA4) compared to standard stock K-quants.
📋 Prerequisites
Before starting, ensure you have an environment configured with AMD ROCm or Vulkan runtimes and the following dependencies installed:
- OS: Linux (Ubuntu 24.04 recommended) or Windows 10/11
- Hardware: AMD Radeon GPU / APU (e.g., Strix Halo Ryzen AI Max, RX 7900 XTX)
- Python: 3.10+ with
piporuv - Disk Space: ~150 GB (To hold the raw weights, intermediate standard GGUF, and final ROCmFPX outputs)
🛠️ Step 1: Environment Setup
Clone the custom ROCmFPX fork of llama.cpp and install its conversion dependencies.
# Clone the ROCmFPX repository
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
# Install required Python packages
pip install -r requirements.txt
# Alternatively using uv for faster installation:
# uv pip install -r requirements.txt --index-strategy unsafe-best-match
📥 Step 2: Download KATCoder-2.5-Dev Weights
Download the original Hugging Face safetensors for the model.
# Ensure huggingface-cli is installed
pip install huggingface_hub
# Download the model weights to a local directory
huggingface-cli download Kwaipilot/KAT-Coder-V2.5-Dev --local-dir ./KAT-Coder-V2.5-Dev
# Download Qwen 3.6 35B A3B for MTP
huggingface-cli download Qwen/Qwen3.6-35B-A3B --local-dir ./Qwen3.6-35B-A3B
🔄 Step 3: Extract MTP from Qwen and Inject into KATCoder and then Convert to Base GGUF Format
Convert the Hugging Face safetensors structure into a standard unquantized 16-bit GGUF base file using the repository's native script.
python graft_mtp.py # cd to the root folder where KATCoder Safetensors and Qwen Safetensors are available and copy graft_mtp.py in that folder and execute
python convert_hf_to_gguf.py ./KAT-Coder-V2.5-Dev-MTP \
--outfile ./KAT-Coder-V2.5-Dev-bf16.gguf \
--outtype bf16
⚡ Step 4: Build the ROCmFPX Binary Compiler
Compile the binaries tailored to your target AMD graphics architecture. Swap the build environment flags depending on your system setup.
For Linux (ROCm / HIP)
cd ROCmFPX && git checkout main
# 2. Build for Strix Halo
env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh
💎 Step 5: Quantize to ROCmFPX GGUF
Run the compiled llama-quantize tool to compress the model into the optimized ROCmFPX layout. Common targets include rocmfp4 (4-bit format) or rocmfp6.
Reference: https://huggingface.co/plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF (But didnt use imatrixi for now)
# General syntax: ./build/bin/llama-quantize [input_f16_gguf] [output_gguf] [quant_type]
./build-strix-rocmfp4-mtp/bin/llama-quantize --token-embedding-type f16 --output-tensor-type q6_K KAT-Coder-V2.5-Dev-MTP-bf16.gguf KAT-Coder-V2.5-Dev-MTP-ROCmFP4-STRIX-LEAN-embedF16-headQ6.gguf Q4_0_ROCMFP4_STRIX_LEAN
(Note: You can swap rocmfp4 for other experimental tiers like rocmfp2 or rocmfp6 depending on your required performance-to-perplexity trade-off).
🚀 Step 6: Inference Run
Once the quantization completes, load and prompt the model directly within the ROCmFPX native CLI layer.
./build/bin/llama-cli \
-m ./KAT-Coder-V2.5-Dev-MTP-ROCmFP4-STRIX-LEAN-embedF16-headQ6.gguf \
-n 512 \
--color \
-p "Create a spaceship game inspired by Star Fox using vanilla Three.js and HTML."
📄 License & Attributions
- Base Model: Developed by Kwaipilot under the Apache 2.0 license.
- Quantization Engine: Created by the ROCmFPX Team building upon the standard
llama.cppcodebase framework.