Gemma 4 E4B Uncensored NVFP4

EAGLE speculative decoding drafter for Gemma 4 26B-A4B Uncensored (TrevorJS) — a 42-layer E4B (EAGLE for Blackwell) model quantized to NVFP4 AWQ using NVIDIA ModelOpt 0.42.0.

Designed for EAGLE-based speculative decoding on NVIDIA DGX Spark (GB10, SM 12.1) and other Blackwell-architecture GPUs.

GitHub Repo — patches, deployment configs, Dockerfile

Model Details

Quick Start

Prerequisites

1. Target model — Any NVFP4-quantized Gemma 4 26B MoE (e.g., AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4)
2. This drafter model — Download below
3. Three vLLM patches — Required for Gemma 4 speculative decoding (see Required Patches)
4. Pre-built container — ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest

1. Download both models

pip install -U huggingface-hub

# Target model (26B MoE)
huggingface-cli download AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/trevorjs-26b

# This drafter model (E4B)
huggingface-cli download AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 \
  --local-dir ~/models/e4b-drafter

2. Get the patched vLLM files

Three patches to vLLM 0.19.1 are required for Gemma 4 speculative decoding. Download from the DECKARD 31B GitHub repo:

for f in eagle_patched.py serving_chat_patched.py modelopt_patched.py; do
  curl -LO https://raw.githubusercontent.com/AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4/main/$f
done

3. Launch with Docker Compose

services:
  vllm:
    image: ghcr.io/aeon-7/vllm-spark-gemma4-nvfp4-awq:latest
    container_name: vllm-trevorjs-26b-spec
    restart: unless-stopped
    network_mode: host
    volumes:
      - ~/models/trevorjs-26b:/models/target
      - ~/models/e4b-drafter:/models/e4b-drafter
      - ./modelopt_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/modelopt.py
      - ./serving_chat_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py
      - ./eagle_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/spec_decode/eagle.py
    environment:
      - VLLM_TEST_FORCE_FP8_MARLIN=1
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - VLLM_USE_FLASHINFER_MOE_FP4=1
      - TORCH_MATMUL_PRECISION=high
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    command:
      - bash
      - -c
      - |
        exec vllm serve /models/target \
          --served-model-name trevorjs-26b \
          --quantization modelopt \
          --dtype auto \
          --kv-cache-dtype fp8 \
          --tensor-parallel-size 1 \
          --max-model-len 131072 \
          --max-num-seqs 4 \
          --gpu-memory-utilization 0.65 \
          --trust-remote-code \
          --host 0.0.0.0 --port 8000 \
          --enable-chunked-prefill \
          --enable-prefix-caching \
          --enable-auto-tool-choice \
          --tool-call-parser gemma4 \
          --reasoning-parser gemma4 \
          --speculative-config '{"method":"draft_model","model":"/models/e4b-drafter","num_speculative_tokens":5,"quantization":"modelopt"}'
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes and stalls the box, and a spec-decode drafter's verify buffers are not counted by the fraction so leave headroom (0.65 here). Discrete-VRAM GPUs can run higher.

Speculative Config Parameters

Required vLLM Patches

Speculative decoding with Gemma 4 requires three patches to vLLM 0.19.1. Without these, the server will crash on startup.

Patch 1: eagle_patched.py — Gemma 4 spec decode support

File: vllm/v1/spec_decode/eagle.py

Three fixes are needed:

1a. Remove multimodal guard

vLLM 0.19.1 calls _raise_if_multimodal() which blocks ALL multimodal targets from speculative decoding, even when the drafter is text-only. Remove this call — the downstream code already handles text-only drafters with multimodal targets correctly.

# In initialize() method — REMOVE this line:
# self._raise_if_multimodal()

1b. Add Gemma4 to model whitelist

Gemma 4 uses image_token_id (258880) but NOT image_token_index. The spec decode framework needs an explicit mapping:

# In the model whitelist check, add Gemma4:
if self.get_model_name(target_model) in [
    "Qwen2_5_VLForConditionalGeneration",
    # ... existing models ...
    "Gemma4ForConditionalGeneration",  # ADD THIS
]:
    self.model.config.image_token_index = target_model.config.image_token_id

1c. Multi-group KV cache support

Gemma 4 uses heterogeneous attention: head_dim=256 for sliding-window layers and head_dim=512 for global attention layers. This creates two distinct KV cache groups. The spec decode framework assumes a single group.

Fix validate_same_kv_cache_group to log a warning instead of asserting, and rewrite initialize_attn_backend to key attention groups by (backend_class, kv_cache_group_id) instead of just backend_class, mapping each draft layer to its correct KV cache group.

Patch 2: serving_chat_patched.py — Non-streaming reasoning parser

File: vllm/entrypoints/openai/chat_completion/serving.py

Gemma 4's reasoning parser uses <|channel> (token 100) and <channel|> (token 101) delimiters. With skip_special_tokens=True (the default for non-streaming), these are stripped, causing extract_reasoning() to return None — thinking content lands in the content field.

The fix re-decodes from raw token_ids with skip_special_tokens=False when text-based extraction fails:

if reasoning is None and token_ids and hasattr(reasoning_parser, 'start_token_id'):
    token_ids_list = list(token_ids)
    if reasoning_parser.start_token_id in token_ids_list:
        full_text = reasoning_parser.model_tokenizer.decode(
            token_ids_list, skip_special_tokens=False
        )
        reasoning, content = reasoning_parser.extract_reasoning(full_text, request=request)
        if content:
            _tok = reasoning_parser.model_tokenizer
            content = _tok.decode(_tok.encode(content), skip_special_tokens=True)

Patch 3: modelopt_patched.py — NVFP4 AWQ support

File: vllm/model_executor/layers/quantization/modelopt.py

Three fixes:

1. FP8 NaN scrubbing — ModelOpt 0.42.0 produces ~60 FP8 NaN values (0x7F/0xFF) in weight_scale tensors. Scrubs to zero at load time.
2. NVFP4_AWQ quant_algo — Registers NVFP4_AWQ (upstream only handles NVFP4).
3. AWQ pre_quant_scale — Loads and applies per-channel pre_quant_scale tensors for AWQ weight redistribution.

Applying the Patches

Mount as volume binds in Docker Compose (shown above), or copy manually:

VLLM_PATH=$(python3 -c "import vllm; print(vllm.__path__[0])")
cp eagle_patched.py    $VLLM_PATH/v1/spec_decode/eagle.py
cp serving_chat_patched.py $VLLM_PATH/entrypoints/openai/chat_completion/serving.py
cp modelopt_patched.py $VLLM_PATH/model_executor/layers/quantization/modelopt.py

Heterogeneous Attention Architecture

This E4B drafter mirrors the Gemma 4 heterogeneous attention design:

• 35 sliding-window layers — head_dim=256, window of 512 tokens, default RoPE (theta=10000)
• 7 full-attention layers — head_dim=512, global attention, proportional RoPE (theta=1M, partial_rotary_factor=0.25)

This creates two distinct KV cache groups within the drafter, handled by the multi-group KV cache fix in eagle_patched.py.

Cross-Model Drafter Compatibility

This drafter was derived from the TrevorJS uncensored fine-tune. It can also be used with other Gemma 4 targets that share the same vocabulary (262K tokens):

Acceptance rate will be highest with the matching base model and lower with mismatched fine-tunes, but all combinations will function correctly.

Related Models

Hardware Requirements

• Target + Drafter combined: ~26 GB (16 GB target + 9.6 GB drafter)
• Recommended: NVIDIA DGX Spark (128 GB unified memory) or any GPU with >= 40 GB VRAM
• Required: Blackwell architecture (SM 10.0+) for native FP4

License

This model inherits the Gemma license from Google.

☕ Support the work

AEON-7 models, drafters and tools are built and trained independently, on my own hardware. If they're useful to you, please consider supporting development:

Become a member on Patreon → patreon.com/cw/AeonForge7/membership

Milestones unlock bigger work: reaching 500 paid supporters will fund fine-tuning larger models and bigger project releases. Supporters also get early access to new releases, such as the AEON DFlash2 drafter.

If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.

Ethereum L2s (Base, Arbitrum, Optimism, Polygon, etc.) and EVM-compatible tokens can be sent to the same Ethereum address.