Uploaded finetuned model

• Developed by: filipwx
• License: apache-2.0
• Finetuned from model : unsloth/gemma-4-E2B-it

This gemma4 model was trained 2x faster with Unsloth and Huggingface's TRL library.

license: apache-2.0 tags:

• gemma4
• vision
• multimodal
• image-to-text
• sharded

Gemma-4 E2B-it Opus 4.6 Reasoning (2GB Sharded & VRAM-Optimized)

This is a locally sharded and VRAM-optimized deployment version of the original filipwx/gemma-4-E2B-it-Opus-4.6-Reasoning repository.

Why was this created?

The original repository hosts the entire model inside a single massive 10.2 GB model.safetensors file. On consumer hardware with lower VRAM pools (like a 4GB NVIDIA RTX 3050 Laptop GPU), loading a single shard causes two critical points of failure:

1. The Meta Tensor Crash: transformers defaults to using device_map="auto", which routinely chokes on a single 10GB shard, offloading critical token embeddings to a placeholder meta device and triggering: RuntimeError: Tensor.item() cannot be called on meta tensors.
2. The Dual-GPU Laptop Bug: On hybrid laptop configurations (Intel/AMD Integrated Graphics + Dedicated NVIDIA GPU), accelerate frequently miscalculates available space, outputs the warning does not fit any GPU's remaining memory, and leaves the model completely unutilized in host System RAM (0% VRAM usage).

The Fix

This repository breaks the weights down into optimized 2 GB safe serialization shards (model-00001-of-00005.safetensors, etc.).

When combined with a direct root device mapping (device_map={"": 0}), the shards completely bypass the broken automated memory-checking loops, allowing the 4-bit quantized layers to anchor directly into your dedicated NVIDIA graphics memory.

Verified Working Implementation

Use this exact configuration block to load the model seamlessly. This bypasses the deprecated torch_dtype warning, forces dtype=torch.bfloat16, and mandates direct GPU execution:

from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
import torch

model_id = "pepsisodayoda/gemma-4-sharded-E2B-it-Opus-4.6-Reasoning"

print("Initializing 4-bit bitsandbytes VRAM routing...")
processor = AutoProcessor.from_pretrained(model_id, fix_mistral_regex=True)

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)

# Hard override targeting the primary NVIDIA card directly ({"": 0})
# This forces the weights out of system RAM and onto the graphics card hardware
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
    device_map={"": 0}, 
    low_cpu_mem_usage=True
)

print("Gemma-4 Shards successfully forced onto Dedicated GPU VRAM!")