Update
Added a Jinja chat template so the model can format conversations correctly and work smoothly with mlx-lm chat-style inference.
MLX 8-Bit Quantized: Gemma-4-12B-Coder
This repository contains an 8-bit MLX-converted version of yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1.
The model has been quantized to 8-bit to dramatically reduce memory requirements while retaining near-lossless reasoning and coding capabilities. It is optimized for local inference on Apple Silicon Macs using the mlx-lm library.
How to Use with MLX
Install the required dependency:
pip install --upgrade mlx-lm
Run inference from Python:
from mlx_lm import load, generate
# Load the 8-bit quantized MLX model.
model, tokenizer = load("mlx-community/gemma-4-12b-coder-fable5-composer2.5-8bit")
prompt = "Write a Python script to sort a dictionary by its values."
messages = [{"role": "user", "content": prompt}]
formatted_prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
response = generate(
model,
tokenizer,
prompt=formatted_prompt,
verbose=True,
max_tokens=1024,
)
response = generate(
model,
tokenizer,
prompt=formatted_prompt,
verbose=True,
max_tokens=1024,
temp=0.0,
)
Base and License
- Base model:
google/gemma-4-12B-it - Original fine-tune:
yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1 - License: Apache 2.0
Free to use, modify, and redistribute under the Apache 2.0 license.