🧠 Optimization: Custom APEX Quants

This repository contains custom APEX (Adaptive Precision for EXpert Models) quants built with the specific mixture-of-experts architecture in mind.

⚠️ Disclaimer: This repository is an independent open-source optimization project. All credit for the underlying architecture, pruning/merging mechanics, and original model weights belongs entirely to the various upstream authors. The evaluation suite was compiled using the imatrix-dataset generated by mradermacher.

• Why APEX? Standard uniform quantization applies compression equally across all layers, which can degrade MoE performance. The APEX layout locks the routing blocks at high-precision and shields vital shared experts, shifting aggressive compression strictly to redundant mid-layer tensors.

Note: The quantization tags listed on the front page (e.g., Q3_K_M) indicate the base quantization level passed to the compiler. The actual internal layer distribution utilizes non-uniform APEX mapping.

The basic APEX package includes:

• i-Mini: 8673.04 MiB (3.94 BPW) — Extremely nimble, highly stable baseline.
• i-Compact: 9587.57 MiB (4.36 BPW) — The all-rounder sweet spot.
• i-Balanced: 13851.23 MiB (6.30 BPW) — Premium layer tracking with mid-expert compression.
• i-Quality: 13000.45 MiB (5.91 BPW) — Advanced Importance Quantization (iq4_xs/nl) valley logic

Additionally included are my personal attempts at creating an intermediary size-tier between Compact and Balanced, by scaling up the basic logic of the Compact configuration by one quant level:

• i-Upscale.Econ: 9541.90 MiB (4.34 BPW) — up_exps, down_exps Q4_K for beginning- and end-layers, Q3_K in middle layers. (From testing: Unfurls into larger overall memory footprint on legacy hardware, average performance.)
• i-Upscale.Balanced: 10871.25 MiB (4.94 BPW) — no mixed-precision gradient exps, just flat Q4_K for all up_exps, down_exps. (From testing: Good known performance/precision/size trade-off.)
• i-Upscale.Premium: 10738.31 MiB (4.88 BPW) — up_exps, down_exps Q5_K for beginning- and end-layers, Q4_K and Q3_K in middle layers. (From testing: Stays small in memory on legacy hardware, best performance of the three.)

Test data not gospel, just quick first impression of the finished quants, your experience may vary. Any kind of thoughts & feedback welcome; make a thread, share your experience.

This is a decensored version of 0xSero/gemma-4-19b-a4b-it-REAP, made using Heretic v1.2.0 with the Arbitrary-Rank Ablation (ARA) method (with row-norm preservation)

Abliteration parameters

Performance

Gemma 4 19B-A4B-it REAP

30% expert-pruned version of google/gemma-4-26b-a4b-it using Cerebras REAP (Router-weighted Expert Activation Pruning).

REAP removes 30% of MoE experts (38 of 128 per layer) while preserving the model's routing behavior. The active parameter count per token is unchanged since the router still selects 8 experts per token from the remaining pool. This yields a ~31% reduction in total disk/memory footprint.

How This Model Was Made

Step 1: Calibration (Activation Observation)

We ran the full Gemma 4 26B-A4B-it model over a curated calibration dataset to record expert activation patterns across all layers.

Calibration dataset: 22,000 samples drawn from 12 sources:

Step 2: REAP Pruning

The lowest-scoring 30% of experts (38 per layer) are removed based on combined router gate values, activation norms, and frequency-weighted saliency. Router logits are renormalized post-pruning.

Pruning Configuration

Benchmark Results

Accuracy (generative, 0-shot, 50 samples/task, thinking enabled, vLLM 0.19, 4x RTX 3090)

Evaluated using lm-eval generative tasks with --apply_chat_template and think_end_token=<channel|> to properly handle Gemma 4's thinking mode.

* Tasks with extraction failures. Real accuracy likely higher.

Summary: At 30% pruning, the model retains strong performance on elementary math (88%) and coding (68% college CS). Knowledge-intensive tasks like philosophy and world religions see moderate drops. The 30% variant trades ~18 pts on philosophy and ~42 pts on world religions for a 31% smaller model.

Robustness: Generation Quality (14 prompts, temp=0.7, top_p=0.9, max 2048 tokens)

Zero looping, zero collapse across coding, math, and philosophy. The REAP 0.30 model actually outperforms the original on long-context (0% vs 50% loop rate). Generation quality is fully preserved at 30% compression.

Architecture

Gemma 4 uses a hybrid sliding/full attention MoE architecture:

• 30 transformer layers
• Sliding attention (window=1024) for 25 layers, full attention every 6th layer
• MoE FFN with 90 remaining experts per layer (originally 128), 8 active per token
• Thinking model -- uses <|channel>thought / <|channel>response channels
• Multimodal -- supports text and vision inputs
• Context window: 262,144 tokens
• Vocab size: 262,144

Usage

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "0xSero/gemma-4-19b-a4b-it-REAP"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True)

messages = [{"role": "user", "content": "Write a quicksort in Python."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=4096)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

vLLM

pip install vllm>=0.19 transformers>=5.0

vllm serve 0xSero/gemma-4-19b-a4b-it-REAP \
    --tensor-parallel-size 2 \
    --enforce-eager \
    --gpu-memory-utilization 0.9 \
    --max-model-len 16384 \
    --trust-remote-code

Citation

@inproceedings{lasby2025reap,
  title={{REAP} the Experts: Why Pruning Prevails for One-Shot {MoE} Compression},
  author={Lasby, Mike and others},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2026},
  url={https://arxiv.org/abs/2510.13999}
}

Links

• REAP paper: arxiv.org/abs/2510.13999
• REAP code: github.com/cerebras/reap
• 20% pruned variant: 0xSero/gemma-4-21b-a4b-it-REAP
• Base model: google/gemma-4-26b-a4b-it