gemma-4-31B-it-FP8-block
This repository contains RedHatAI/gemma-4-31B-it-FP8-block together with a Furiosa Executable Bundle (FXB) for running it on FuriosaAI RNGD with Furiosa-LLM. The same model also runs on other frameworks (such as vLLM, SGLang, and Transformers); for usage with those, see the upstream RedHatAI/gemma-4-31B-it-FP8-block model card.
Overview
Gemma 4 31B is a dense multimodal model from Google. It combines a vision encoder with a transformer decoder that interleaves sliding-window and full-attention layers. Its chat template supports optional reasoning and tool (function) calling. This Red Hat AI build applies FP8 block quantization to the text transformer's linear layers. The current Furiosa FXB supports text-to-text inference; image input is not yet supported. Its intended use is the same as the upstream RedHatAI/gemma-4-31B-it-FP8-block, released under the Apache 2.0 License.
- Architecture: Gemma 4 (dense)
- Input / Output: Text / Text with the current FXB (the upstream model also supports image + text input)
- Supported Inference Engine: Furiosa LLM
- Supported Hardware: FuriosaAI RNGD
Quantization
The upstream Red Hat AI build statically quantizes the weights of linear operators in the text transformer to FP8 with 128×128 blocks. Their input activations are quantized dynamically to FP8 at runtime in groups of 128 values, without offline calibration. The vision tower, embeddings, output head, and KV cache stay in 16-bit precision.
Features
- Hybrid thinking. The chat template honors
enable_thinkingand defaults it tofalse. The current Furiosa-LLM release does not provide a Gemma 4 reasoning parser, so the usage below keeps thinking disabled. - Vision-language architecture. The upstream model accepts image and text input. The current Furiosa FXB supports text input only.
- Tool calling. The upstream model has a native tool-call format. The current Furiosa-LLM release does not provide a compatible tool-call parser.
Parallelism Strategy
On RNGD, gemma-4-31B-it-FP8-block runs with a tensor-parallel size of 32 PEs, which maps to four RNGD cards (8 PEs per card).
Usage
To run this model with Furiosa-LLM, follow the example commands below after installing Furiosa-LLM and its prerequisites.
Launch the server
Serve the model by passing its furiosa-ai/<repo> identifier:
# Launch the server, listening on port 8000 by default
furiosa-llm serve furiosa-ai/gemma-4-31B-it-FP8-block
Gemma 4's chat template can generate reasoning and tool calls, but Furiosa-LLM does not currently register parsers for the Gemma 4 output format. Keep the default non-thinking chat mode and do not enable automatic tool choice with this FXB.
When the server is ready, you will see:
INFO: Started server process [27507]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Basic Usage
The server exposes an OpenAI-compatible API. You can send a text request with
curl:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "furiosa-ai/gemma-4-31B-it-FP8-block",
"messages": [{"role": "user", "content": "What is the capital of France?"}]
}' \
| python -m json.tool
Advanced Usage
Current feature scope. The upstream model accepts image input and its chat
template supports optional reasoning through enable_thinking as well as native
tool calls. The current Furiosa FXB supports text input only, and Furiosa-LLM
does not yet provide the Gemma 4 reasoning and tool-call parsers required to
return those outputs through the corresponding OpenAI-compatible response
fields. Multimodal serving flags and automatic tool choice therefore do not
enable these upstream capabilities for this FXB.
See the Vision-Language Models guide and Tool Calling guide for the APIs that will apply when those capabilities are supported.
Learn more
- Vision-Language Models — image input formats and multimodal server options
- Tool Calling — parsers, tool-choice options, and examples
- Furiosa-LLM Server (
furiosa-llm serve) — full OpenAI-compatible API reference and serving options - RedHatAI/gemma-4-31B-it-FP8-block — upstream quantized model card
- google/gemma-4-31B-it — base model card