Quatfit-Mini-FP8

Overview

This repository provides an FP8-quantized build of Quatfit Mini, an 8B-parameter multimodal model built on the Gemma 4 architecture. It is intended for high-throughput, GPU-served inference on FP8-capable hardware (NVIDIA Hopper, Ada Lovelace, and Blackwell generations), where it delivers close to BF16 quality at roughly half the memory footprint and with hardware-accelerated FP8 tensor-core throughput.

Unlike the GGUF builds — which target CPU and consumer-GPU local inference via llama.cpp — this repository is designed for server-side deployment with runtimes such as vLLM, TensorRT-LLM, and SGLang.

For local inference on consumer GPUs or CPU, use Quatfit/Quatfit-Mini-GGUF instead. For fine-tuning or maximum numerical fidelity, use the base Quatfit/Quatfit-Mini (FP32) repository.

Quantization Recipe

Following the same reasoning applied to the GGUF mmproj files, the vision and audio encoders are more sensitive to aggressive quantization than the text backbone, so they are left in BF16 here rather than cast to FP8. Only the text decoder's linear layers — the overwhelming majority of parameters and compute — are quantized, which is where FP8's throughput and memory benefits matter most.

Model Summary

Why FP8

FP8 differs from the quantized GGUF builds in an important way: on supported hardware, FP8 isn't just a smaller file — it runs on native FP8 tensor cores, so it increases raw compute throughput in addition to cutting memory and bandwidth, rather than only reducing memory like a CPU-oriented integer quant. Relative to the BF16-cast version of Quatfit Mini on the same GPU:

Exact gains depend on GPU generation, batch size, and sequence length; the ranges above reflect typical serving conditions rather than a specific benchmark run.

Hardware Requirements

FP8 tensor-core acceleration requires one of the following NVIDIA architectures:

• Hopper — H100, H200
• Ada Lovelace — L4, L40, L40S, RTX 4090, RTX 4080
• Blackwell — B100, B200, RTX 50-series

On GPUs without native FP8 support (e.g. Ampere: A100, A10, RTX 3090), most runtimes will either reject FP8 weights or fall back to upcasting them to BF16 at load time — memory savings are retained, but the throughput benefit of native FP8 compute is not. For Ampere-class or older hardware, casting the base FP32 model to BF16 directly, or using a GGUF build, is generally a better fit.

Usage

vLLM

vllm serve Quatfit/Quatfit-Mini-FP8 \
  --max-model-len 131072 \
  --quantization fp8

from vllm import LLM, SamplingParams

llm = LLM(model="Quatfit/Quatfit-Mini-FP8", quantization="fp8", max_model_len=131072)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

outputs = llm.generate(["Explain the difference between GQA and MHA."], sampling_params)
print(outputs[0].outputs[0].text)

TensorRT-LLM

trtllm-build \
  --checkpoint_dir Quatfit-Mini-FP8-checkpoint \
  --output_dir ./trt_engines/quatfit-mini-fp8 \
  --gemm_plugin fp8 \
  --max_input_len 131072

🤗 Transformers

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini-FP8",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("Quatfit/Quatfit-Mini-FP8")

messages = [{"role": "user", "content": "Write a Python implementation of binary search."}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(outputs[0]))

On GPUs without native FP8 tensor cores, Transformers will run this checkpoint by upcasting to BF16 automatically — you'll still get the ~9.5 GB memory footprint benefit, but not the FP8 compute speedup.

Prompt Format

Quatfit Mini uses the Gemma 4-style turn format with support for tool calling and an optional thinking mode:

<|turn>user
Your message here<turn|>
<|turn>model

To enable extended reasoning traces, prefix the conversation with a system turn containing <|think|>. The model will emit its reasoning inside <|channel>thought ... <channel|> before the final response.

Benchmark Scores

FP8 scores match the FP32 reference weights within noise (< 0.3 points on average across categories), consistent with the calibrated, static-scale quantization recipe used here.

Relationship to Other Quatfit Mini Repositories

Full architecture details, training recipe, and benchmark methodology are documented in the Quatfit Mini Technical Report.

Responsible AI

Quatfit Mini may generate inaccurate, biased, or inappropriate outputs. FP8 quantization of the text decoder introduces a small amount of additional numerical noise relative to the FP32/BF16 reference, though evaluation shows this has negligible practical effect. For production deployments:

• Verify critical information
• Apply RAG for factual grounding
• Use application-level safety filters
• Keep human oversight for high-risk domains

Citation

@article{quatfitmini2026,
    title={Quatfit Mini: A Gemma 4-Based Multimodal Model Optimized for Efficient Inference},
    author={Quatfit AI Research},
    year={2026}
}

License

Apache License 2.0. See LICENSE for details.