gemma-4-E4B-it · pruned 30% & distilled with QAT, packed in W4A16

A compute-and-energy-optimized google/gemma-4-E4B-it built by a five-stage pipeline:

1. Structural MLP prune (−12.5%) — the LM-stack gate_proj/up_proj/down_proj intermediate dimension is reduced 12.5% with a calibrated importance criterion. Vision/audio towers and attention are untouched.
2. Knowledge distillation with QAT — the pruned LM is recovered toward the unpruned bf16 teacher (Phase 1 forward-KL + hidden-state matching, Phase 2 on-policy JSD for brevity) using fake-quantization-in-the-loop. Vision/audio towers frozen. This step aims to restore both capability and output brevity of the student model.
3. Second MLP prune (−20%) — a second-stage pruning is applied to arrive at an effective 30% pruning compared to base model. Same recipe as first-stage pruning.
4. Second KD with QAT — recovers the 30%-pruned LM towards base model.
5. W4A16 quantization — int4 weight-only quantization via llmcompressor oneshot + QuantizationModifier (observer-only; no Hessian/AWQ). Activations stay bf16. Uses the exact same quantization recipe as the KD with QAT steps to avoid drift.

Saved in compressed-tensors pack-quantized format — loads in HF Transformers (Marlin / GPTQ-Marlin kernels, run_compressed=True) and in vLLM via the CompressedTensorsWNA16 loader.

Quantization recipe

QuantizationModifier(
    config_groups={
        "group_0": {
            "targets": ["Linear"],
            "weights": {
                "num_bits": 4,
                "type": "int",
                "symmetric": True,
                "strategy": "group",
                "group_size": 128,
                "observer": "minmax",
                "actorder": None,
                "dynamic": False,
            },
            "input_activations": None,
            "output_activations": None,
        }
    },
    ignore=[
        "re:.*vision_tower.*",          # ViT encoder + patch embedder
        "re:.*audio_tower.*",           # audio layers + subsample + output_proj
        "re:.*per_layer_input_gate.*",  # PLE input gates
        "re:.*per_layer_projection.*",  # PLE projections
        "re:.*embed_vision.*",          # vision embedding_projection
        "re:.*embed_audio.*",           # audio embedding_projection
        "lm_head",
    ],
)

Only LM-stack Linear weights are packed to int4. The vision tower, audio tower, Per-Layer Embedding (PLE) plumbing, vision/audio projectors, and lm_head stay bf16.

Inference

vLLM

No quantization= argument — vLLM auto-detects compressed-tensors from config.json and binds to MarlinLinearKernel.

vllm serve terra-cognita-ai/ResAI_Image-to-Text_Champ --config vllm_config.yaml

The vllm_config.yaml is included in the root directory of the model.

HF Transformers

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch

ckpt = "<this repo>"   # or local path
model = AutoModelForImageTextToText.from_pretrained(
    ckpt, dtype=torch.bfloat16, device_map="cuda:0",
)
processor = AutoProcessor.from_pretrained(ckpt)

Pass run_compressed=True to keep packed int4 weights in VRAM through the forward pass (no on-the-fly unpack to bf16).

Limitations

• Ada attention fallback. vLLM on Ada GPUs (L4 / RTX 4090) takes the Triton attention path for this model regardless of quantization (vLLM #38887) — a constant-factor latency hit shared with the bf16 baseline. Hopper/Ampere take the FlashAttention path.

License

Inherits the Gemma License from the base model. By using this checkpoint you agree to the Gemma Terms of Use.

Acknowledgements

• vllm-project/llm-compressor — oneshot + QuantizationModifier.
• neuralmagic/compressed-tensors — Marlin int4 kernels.
• optipfair — the calibrated maw_hybrid pruning criterion.