Qwen3.8-27B-GSQ-RCO-MLX
An Apple Silicon MLX package of the mixed-IQ GSQ-RCO Qwen3.8-27B checkpoint.
The original GGUF quantized tensor payloads are preserved losslessly inside a
packed Safetensors file. This is a custom packed format, not a standard
mlx-lm checkpoint. Load it with the separate
mlx-GSQ-RCO runtime.
Files
model.safetensors: packed payloads for 866 tensors (about 9.7 GiB)model.safetensors.manifest.json: tensor shapes and quantization metadata- Tokenizer and model configuration files
The weights use 11 quantized types: IQ1_M, IQ1_S, IQ2_XXS, IQ2_XS,
IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q2_K, Q4_K, and Q6_K. Plain F32
and BF16 tensors are also present. The GGUF-to-Safetensors converter verifies
all 866 source payloads by SHA-256.
Requirements and usage
- Apple Silicon Mac with Metal
- Python 3.10+
- About 10.4 GB unified memory for a minimal generation run; extra headroom is recommended
Install the runtime and generate text (the first run downloads this model):
python -m pip install \
'mlx-gsq-rco[mlx,integration] @ git+https://github.com/uqer1244/mlx-GSQ-RCO.git'
mlx-gsq-generate \
uqer1244/Qwen3.8-27B-GSQ-RCO-MLX \
--prompt "Hello" \
--max-tokens 32
Python API:
from mlx_gsq import load
from mlx_lm.generate import generate
model, tokenizer = load("uqer1244/Qwen3.8-27B-GSQ-RCO-MLX")
print(generate(model, tokenizer, prompt="Hello", max_tokens=32))
Recorded performance
On an Apple M3 Pro, the runtime measured 22.3 tok/s prefill for a 182-token prompt and 9.14 tok/s for 64-token decode. A 256-token generation run used about 10.3 GB peak unified memory. The half4-vectorized tiled QMM kernel improved large single-layer QMM throughput by 2.6–3.2x versus the earlier scalar GEMM version; end-to-end prefill improved from 10.5 to 22.3 tok/s in the recorded run. Results vary by machine, prompt, and software versions. More benchmark context is in the GitHub README.
Limitations
- Text generation only; vision inference is not implemented.
- The runtime currently targets this Qwen3.8/Qwen3.5 hybrid architecture.
mlx_lm.load()cannot load this custom checkpoint directly; usemlx_gsq.load().
License and attribution
The model weights are distributed under the upstream Apache-2.0 license and
retain their original attribution. The runtime code is MIT licensed. The
original mixed-IQ GGUF is
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF.