gemma-4-31B-it-GGUF
[!NOTE] Read our blog post to learn more about using Gemma 4 with
node-llama-cpp
Static quants of google/gemma-4-31B-it.
Quants
| Link | URI | Quant | Size |
|---|---|---|---|
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q2_K | Q2_K | 11.9GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q3_K_S | Q3_K_S | 13.8GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q3_K_M | Q3_K_M | 15.3GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q3_K_L | Q3_K_L | 16.6GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q4_0 | Q4_0 | 17.7GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q4_K_S | Q4_K_S | 17.8GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q4_K_M | Q4_K_M | 18.7GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q5_0 | Q5_0 | 21.3GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q5_K_S | Q5_K_S | 21.3GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q5_K_M | Q5_K_M | 21.8GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q6_K | Q6_K | 25.2GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:Q8_0 | Q8_0 | 32.6GB |
| GGUF | hf:giladgd/gemma-4-31B-it-GGUF:BF16 | BF16 | 61.4GB |
[!TIP] Download a quant using
node-llama-cpp(more info):npx -y node-llama-cpp pull <URI>
Usage
Use with node-llama-cpp (recommended)
Ensure you have node.js installed:
brew install nodejs
CLI
Chat with the model:
npx -y node-llama-cpp chat hf:giladgd/gemma-4-31B-it-GGUF:Q4_K_M
Code
Use it in your project:
npm install node-llama-cpp
import {getLlama, resolveModelFile, LlamaChatSession} from "node-llama-cpp";
const modelUri = "hf:giladgd/gemma-4-31B-it-GGUF:Q4_K_M";
const llama = await getLlama();
const model = await llama.loadModel({
modelPath: await resolveModelFile(modelUri)
});
const context = await model.createContext();
const session = new LlamaChatSession({
contextSequence: context.getSequence()
});
const q1 = "Hi there, how are you?";
console.log("User: " + q1);
const a1 = await session.prompt(q1);
console.log("AI: " + a1);
[!TIP] Read the getting started guide to quickly scaffold a new
node-llama-cppproject
Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
brew install llama.cpp
CLI
llama-cli -hf giladgd/gemma-4-31B-it-GGUF:Q4_K_M -p "The meaning to life and the universe is"
Server
llama-server -hf giladgd/gemma-4-31B-it-GGUF:Q4_K_M -c 2048