FastVLM CPU Inference (Vision Projector + Qwen2 GGUF)
A complete, CPU-only FastVLM inference pipeline: an ONNX vision projector for image encoding, paired with a quantized Qwen2 GGUF language model for text generation. This repo includes the model weights plus the full loading/serving code, so you can run the pipeline end to end without a GPU.
Pipeline overview
Image → [ONNX Vision Projector] → image embeddings ↓ [fastvlm_server (llama.cpp-based C binary)] ↓ Streamed text response (Qwen2 GGUF, Q4_K_M)
- An image is preprocessed and passed through
vision_projector_v1_standalone.onnx(ONNX Runtime, CPU) to produce image embeddings. - The embeddings are written to a temporary binary file and handed to
a persistent
fastvlm_serverprocess (a custom llama.cpp-based binary) that keeps the GGUF language model loaded in memory. - The language model generates a response token-by-token, streamed back over HTTP.
Files in this repo
| File | Role |
|---|---|
vision_projector_v1_standalone.onnx | Vision encoder — converts image → embeddings |
fastvlm_qwen2_q4km.gguf | Quantized Qwen2-0.5B language model (Q4_K_M) |
fastvlm_server | Compiled C binary (llama.cpp-based) that loads the GGUF model and serves generation over stdin/stdout |
libggml.so, libggml-base.so, libggml-cpu.so, libllama.so (+ .0 versions) | Shared libraries required by fastvlm_server at runtime |
stream_api.py | FastAPI server tying the vision encoder and LLM server together, exposing /predict |
Requirements
- Python 3.9+
onnxruntime,fastapi,uvicorn,pillow,numpy,python-multipart
pip install onnxruntime fastapi uvicorn pillow numpy python-multipart
Loading and running
Download all files in this repo into one local directory, keeping
them all together (stream_api.py looks for the .onnx, .gguf,
fastvlm_server, and .so files relative to its own location), then:
python stream_api.py
This starts a FastAPI server on http://0.0.0.0:8000. On startup it:
- loads the ONNX vision encoder via ONNX Runtime (CPU),
- launches
fastvlm_serveras a persistent subprocess with the GGUF model loaded and kept in memory, so the model isn't reloaded per request.
Endpoints
GET /— basic status infoGET /health— reports whether the ONNX session, GGUF file, and server binary are all present and loadedPOST /predict— send an image + optional prompt, get a streamed text response
Example request:
curl -X POST http://localhost:8000/predict \
-F "image=@your_image.jpg" \
-F "prompt=Describe this image in detail." \
--no-buffer
The response streams as plain text until generation completes.
How it stays fast
fastvlm_server is started once at API startup and kept alive as a
long-running subprocess — the GGUF model is loaded into memory a
single time. Each /predict request sends the embedding file path and
prompt over the process's stdin and reads generated tokens back from
stdout until a ---END--- sentinel, avoiding reload cost per request.
Requests are serialized (one at a time), since the underlying server
is single-threaded.
Eval results
Benchmarks were run on a 6-core Intel CPU, comparing the two input resolution variants of the vision pipeline (512×512 vs 1024×1024).
| Resolution | Visual tokens | TTFT | Benchmark | Score |
|---|---|---|---|---|
| 512×512 | 64 | ~1.4 s | POPE | 80.39% |
| 512×512 | 64 | ~1.4 s | GQA | 60.12% |
| 1024×1024 | 256 | ~5.5–9.6 s | TextVQA | 72.21% |
| 1024×1024 | 256 | ~5.5–9.6 s | OCRBench v2 | 45.57% |
LLaVA-Wild average TTFT @ 512: 1,382 ms (6-core Intel CPU)
Notes
- 512×512 uses fewer visual tokens (64) and is significantly faster to first token, making it a good fit for latency-sensitive use cases and general VQA-style tasks (POPE, GQA).
- 1024×1024 uses more visual tokens (256) for finer-grained visual detail, at the cost of higher TTFT — better suited for tasks requiring precise text/detail recognition (TextVQA, OCRBench v2).
- TTFT (time to first token) figures are CPU-only, measured on a 6-core Intel CPU, and will vary with hardware.
Base models
Derived from:
- apple/FastVLM-0.5B (vision projector)
- Qwen/Qwen2-0.5B (language backbone, quantized to GGUF)
Related
A hosted Gradio demo of this pipeline is available at: musk12/FastVLM-CPU-Inference-demo