Encoder-only extraction for google/t5gemma-2-270m-270m
This repo contains the extracted encoder-only weights including visioning for google/t5gemma-2-270m-270m. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.
Reference table
This extraction is part of a complete T5Gemma suite (270m, 1b, 2b, 4b, 9b) created to provide a unified environment for LLM-SDXL adapter research.
| Model | T5Gemma Series | Architecture | Model Size (GB) | Saved VRAM | Note |
|---|---|---|---|---|---|
| 270m-270m | V2 (Gemma-3) | Multimodal | 1.37 | ~13% | Vision tower takes up most of the full model |
| 1b-1b | V2 (Gemma-3) | Multimodal | 2.84 | ~33% | |
| 2b-2b | V1 (Gemma-2) | Text-only | 5.23 | ~53% | Most VRAM saved of V1 |
| 4b-4b | V2 (Gemma-3) | Multimodal | 8.60 | ~43% | Most VRAM saved of V2 |
| 9b-2b | V1 (Gemma-2) | Text-only | 18.46 | ~25% | Encoder takes up most of the full model |
Why?
Standard T5-Gemma checkpoints are designed for seq2seq tasks, carrying heavy decoder overhead. This extraction isolates the V2 multimodal encoder (Gemma-3 based), stripping the decoder layers to reduce VRAM consumption by ~13%.
How to load the model
from transformers import T5Gemma2Model
model_path = "PhatcatDK/t5gemma-2-270m-270m-encoder-only"
# Load the weights
model = T5Gemma2Model.from_pretrained(
model_path,
torch_dtype="auto", # Recommended: bfloat16
is_encoder_decoder=False, # CRITICAL: Tells Transformers there is no decoder
trust_remote_code=True
)
Acknowledgements & Inspiration
This work was directly inspired by the pioneering research of Minthy.
Minthy was the first to demonstrate the power of the t5gemma-2b-2b-ul2 encoder within this ecosystem. This specific repository was created to round out a standardized suite (270m through 9b) with a "clean-config" setup, ensuring consistent architectural parity for cross-attention testing.
I am incredibly grateful for the groundwork Minthy laid; this suite exists to support and verify the technical infrastructure he built.