gemma-4-12B-it-qat-assistant-bf16
This repository contains Multi-Token Prediction (MTP) drafter weights split from google/gemma-4-12B-it-qat-q4_0-unquantized-assistant for use with mlx-vlm speculative decoding.
This is not a standalone chat or text-generation model. Load it as the draft model alongside a compatible Gemma 4 12B target checkpoint.
Use with mlx-vlm
uv run mlx_vlm.generate \
--model google/gemma-4-12B-it \
--draft-model mlx-community/gemma-4-12B-it-qat-assistant-bf16 \
--draft-kind mtp \
--prompt "Describe this image." \
--max-tokens 256
For local weights:
uv run mlx_vlm.generate \
--model /path/to/target-model \
--draft-model /path/to/gemma-4-12B-mtp \
--draft-kind mtp \
--prompt "Describe this image." \
--max-tokens 256
Model Details
- Model type:
gemma4_unified_assistant - Target architecture: Gemma 4 12B
- Precision: bf16
- Runtime: MLX /
mlx-vlm - Format: Safetensors with MLX-compatible config and tokenizer files
The stored tensors are bf16 MLX-compatible drafter weights.
Intended Use
Use this repo only as a speculative decoding drafter for compatible Gemma 4 12B checkpoints. The target model verifies drafted tokens, while this MTP model proposes candidate tokens per decoding step.
Limitations
This checkpoint requires runtime support for Gemma 4 MTP draft models in mlx-vlm. Standard standalone generation through generic Transformers APIs is not expected to work with this repository by itself.
Please refer to the upstream google/gemma-4-12B-it model card and license terms for model usage constraints.