Unified Multimodal Understanding and Generation on Mobile Device
📌 Overview
Mobile-O-1.5B is the larger variant of the Mobile-O family — a unified vision–language–diffusion model that performs both multimodal understanding (VQA, OCR, reasoning) and image generation within a single architecture. It offers improved generation quality and understanding accuracy over the 0.5B variant.
| Spec | Detail |
|---|---|
| Total Parameters | 3.5B |
| VLM Backbone | FastVLM-1.5B (FastViT + Qwen2-1.5B) |
| Diffusion Decoder | SANA-1600M-512 (Linear DiT + VAE) |
| Connector | Mobile Conditioning Projector |
| Image Resolution | 512×512 |
🎯 Supported Tasks
| Task | Input → Output |
|---|---|
| 💬 Conversational AI | Text → Text |
| 👁️ Image Understanding | Image + Text → Text |
| 🖼️ Image Generation | Text → Image |
| ✏️ Image Editing | Image + Text → Image |
🚀 Quick Start
Download
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="Amshaker/Mobile-O-1.5B",
repo_type="model",
local_dir="checkpoints"
)
Image Understanding
python infer_und.py \
--model_path checkpoints/ \
--image_path assets/cute_cat.png \
--prompt "What is in the image?"
Image Generation
python infer_gen.py \
--model_path checkpoints/ \
--prompt "A vibrant tropical rainforest scene with a scarlet macaw perched on a moss-covered branch"
Image Editing
python infer_edit.py \
--model_path checkpoints/ \
--image_path assets/cute_cat.png \
--prompt "Make the cat wear a hat"
🏗️ Architecture
Mobile-O consists of three main components:
- Vision-Language Model (VLM): FastVLM-1.5B — FastViT vision encoder + Qwen2-1.5B language backbone
- Diffusion Decoder: SANA-600M-512 — lightweight linear DiT with VAE for 512×512 generation
- Mobile Conditioning Projector (MCP): ~2.4M param connector using layerwise feature fusion with temperature-scaled weights, depthwise-separable 1D convolutions, and efficient channel attention
🏋️ Training
Trained in three stages:
- Pre-training — Cross-modal alignment on 9M text-image pairs
- SFT — Supervised fine-tuning on ~105K curated pairs
- Post-training — Unified multimodal training on ~105K quadruplets
🔗 Related Resources
| Resource | Link |
|---|---|
| 🤗 Mobile-O-0.5B | Model |
| 🤗 Mobile-O-0.5B-iOS | iOS Components |
| 📱 iOS App Source Code | Mobile-O-App |
📄 Citation
@article{shaker2026mobileo,
title={Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device},
author={Shaker, Abdelrahman and Heakl, Ahmed and Muhammad, Jaseel and Thawkar, Ritesh and Thawakar, Omkar and Li, Senmao and Cholakkal, Hisham and Reid, Ian and Xing, Eric P. and Khan, Salman and Khan, Fahad Shahbaz},
journal={arXiv preprint arXiv:2602.20161},
year={2026}
}
⚖️ License
Released under CC BY-NC 4.0. For research purposes only.