gemma-3-4b-it-finetome
A supervised fine-tune of google/gemma-3-4b-it on a 12,000-sample high-quality subset of mlabonne/FineTome-100k, trained with QLoRA and merged back into full bf16 weights.
Training details
- Method: QLoRA (4-bit NF4 base, LoRA r=16, alpha=32, dropout 0.05) on all attention + MLP projections of the language model (vision tower untouched), merged to bf16 after training.
- Data: 12k conversations sampled (seed 42) from FineTome-100k, a quality-filtered subset of arcee-ai/The-Tome.
- Schedule: 1 epoch, effective batch size 16, lr 2e-4 cosine with 3% warmup, max seq len 2048.
- Stack: transformers + TRL SFTTrainer + PEFT + bitsandbytes, trained on a single RTX 5090.
Usage
from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch
model_id = "Jkatdare/gemma-3-4b-it-finetome"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "Explain LoRA fine-tuning in one paragraph."}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=300)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
License
Gemma is provided under and subject to the Gemma Terms of Use. This model is a derivative work of Gemma 3 and the same terms apply.