nutrimama-vision
A Gemma 4 12B vision model finetuned to identify Nigerian foods from images.
Model
Base: unsloth/gemma-4-12b-it
HuggingFace: yusasif/nutrimama-vision (branch v2-12b-q8)
Task: Given a food image, name the exact Nigerian dish(es) visible.
The model is finetuned with LoRA (r=32) on vision and language layers using Unsloth for memory-efficient training. The final checkpoint is merged to float16 and optionally quantized to 8-bit for serving.
Dataset
~600 image-text pairs spanning 165+ Nigerian food classes — including:
- Swallows: Eba, Iyan (pounded yam), Fufu, Amala, Tuwon Shinkafa, Tuwon Masara, Nni Ede
- Soups & stews: Egusi, Ogbono, Efo Riro, Gbegiri, Ewedu, Afang, Edikang Ikong, Banga, Ofe Nsala, Okra, Bitterleaf, Ofe Oha, Miyan Taushe, and more
- Rice dishes: Jollof rice, Ofada rice, Fried rice, Enugu Jollof rice
- Beans: Ewa Aganyin, Akara, Moi Moi, Okpa, Kosai
- Snacks & street food: Suya, Kilishi, Dodo, Dundun, Boli, Puff Puff, Kuli Kuli
- Northern dishes: Dambun Shinkafa, Funbau, Miyan Geda, Kunun Aya, Kunun Tsamiya
- Igbo specialties: Abacha, Nkwobi, Ofe Owerri, Ugba Ukpaka, Ukwa, Achicha Ede
- Fruits & drinks: Agbalumo, Nono, Kunun Gyada
Images were collected from multiple dataset batches (nutrimama_dataset, nutrimama_dataset-1, nutrimama_dataset-2, nutrimama_dataset-3) and enriched with detailed Gemma-generated captions (finetune_dataset_gemma.json).
Training
| Setting | Value |
|---|---|
| Base model | unsloth/gemma-4-12b-it |
| LoRA rank | 32 |
| LoRA alpha | 32 |
| Target modules | all-linear (vision + language layers) |
| Epochs | 5 |
| Learning rate | 1e-5 |
| LR scheduler | cosine |
| Batch size | 1 × 8 grad accum = effective 8 |
| Optimizer | adamw_8bit |
| Precision | bfloat16 |
| Max sequence length | 2048 |
Training was run on a single GPU using Unsloth's 4-bit quantized loading with gradient checkpointing.
Inference
from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
MODEL_PATH = "yusasif/nutrimama-vision" # or local path to merged model
processor = AutoProcessor.from_pretrained(MODEL_PATH)
model = AutoModelForImageTextToText.from_pretrained(
MODEL_PATH,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
image = Image.open("your_food_image.jpg").convert("RGB")
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Identify the exact Nigerian food in this image by name."},
{"type": "image", "image": image},
],
}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=input_text, images=[image], return_tensors="pt").to("cuda")
with torch.no_grad():
out = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=1.0,
top_p=0.95,
top_k=64,
)
answer = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)
Serving (vLLM)
python gemma-vision.py # starts an OpenAI-compatible API on port 8002
The server loads the v2-12b-q8 branch and exposes the model as gemma-4-finetuned.
Files
| File | Description |
|---|---|
nutrimam.py | Training script (12B model) |
nutrimam_v2.py | Training script (4B model) |
gemma-vision.py | vLLM inference server |
test_nutrimama.py | Local inference test |
dataset_download_registry.json | Primary image-caption dataset (~398 records) |
finetune_dataset_gemma.json | Gemma-enriched captions (~202 records) |
efficientnet_labels.json | Per-image food class labels |
Related Projects
yusasif/Nigerian-language-tts— multilingual TTS for Nigerian languagesyusasif/nutrimama-vision-v2— 4B model variant