Gemma-4 E2B (5.1B) INT4 OpenVINO
Uncensored Gemma-4 E2B exported to OpenVINO IR with INT4 weight compression via NNCF.
The E2B variant is Gemma-4's efficient architecture with shared KV layers and smaller hidden dimensions, optimized for long-context scenarios.
Model Details
| Property | Value |
|---|---|
| Base model | TrevorJS/gemma-4-E2B-it-uncensored |
| Architecture | Gemma4ForConditionalGeneration |
| Precision | INT4 (asymmetric, group_size=128) |
| Parameters | 5.1B |
| Hidden size | 1536 |
| Layers | 35 |
| Attention heads | 8 |
| KV heads | 1 |
| KV shared layers | 20 |
| Max position | 131072 |
| Sliding window | 512 |
| Vocab size | 262144 |
| Vision | ✅ 280 soft tokens |
| Audio | ✅ |
| Disk size | 4.3 GB |
| Framework | OpenVINO 2026.4+ |
Usage
With openvino_genai (recommended)
import openvino_genai as g
pipe = g.VLMPipeline("gemma-4-E2B-int4-ov", "GPU")
result = pipe.generate("Hello", max_new_tokens=100)
print(result.texts[0])
Multimodal example
result = pipe.generate(
"Describe this image",
images=["image.jpg"],
max_new_tokens=256,
)
print(result.texts[0])
OpenAI-compatible API server
OV_MODEL=./models_trevorjs_e2b_int4_ov_new OV_PORT=8092 python serving/ov_server.py
Performance (Arc A770 16GB, KV_CACHE=f16)
| Context | Latency | Throughput |
|---|---|---|
| Short | 3.5ms | ~285 tok/s |
| 1K | 24.7ms | ~40 tok/s |
| 4K | 28.0ms | ~36 tok/s |
| 8K | 35.8ms | ~28 tok/s |
| 12K | 15.7ms | ~64 tok/s |
| 14K | 46.8ms | ~21 tok/s |
Best value: E2B achieves the best price/performance ratio. With only 4.3GB VRAM, it supports up to 14K context on a 16GB GPU.
Why E2B?
The E2B variant uses 20/35 shared KV layers and 1 KV head to dramatically reduce KV cache memory. This allows longer context windows on limited VRAM compared to the 12B model.
Export Process
Same as other Gemma-4 OpenVINO exports:
- Export FP16 via
optimum-cli export openvino --task image-text-to-text --weight-format fp16 - Compress to INT4 with
nncf.compress_weights(INT4_ASYM, group_size=128, ratio=1.0)
License
Apache 2.0