Nebium-Medium-GGUF
Quantized and FP16 GGUF format binaries for Nebium-Medium (345M parameters).
Designed for low-latency CPU and GPU execution with llama.cpp and Ollama.
- PyTorch Base Model: nabin2004/nebium-medium
- GGUF Repository: nabin2004/nebium-medium-gguf
- Source Repository: github.com/nabin2004/nebium
Artifacts
| Filename | Precision | Description |
|---|---|---|
nebium-medium.gguf | FP16 | Full-precision baseline export |
tokenizer.json | Tokenizer | BPE vocabulary and merge definitions |
Inference with llama.cpp
# Clone and compile llama.cpp
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make
# Download GGUF binary
huggingface-cli download nabin2004/nebium-medium-gguf nebium-medium.gguf --local-dir .
# Run prompt continuation
./llama-cli -m nebium-medium.gguf -p "e2e4 e7e5 g1f3" -n 25 --temp 0.7
Inference with Ollama
# Modelfile
FROM ./nebium-medium.gguf
PARAMETER temperature 0.7
PARAMETER stop "<|eos|>"
SYSTEM You are an autoregressive chess next-move prediction model using UCI move notation.
ollama create nebium-medium -f Modelfile
ollama run nebium-medium "e2e4 e7e5"
License
MIT License.