phonellm-alpha-1 — GGUF
Unofficial GGUF quantizations of pipecat-ai/phonellm-alpha-1.
⚠️ Disclaimer: This is a community conversion. This repository is not affiliated with, endorsed by, or maintained by Pipecat AI or NVIDIA. All credit for the model goes to the original authors — please see the official model card for full details, intended use, and limitations.
About the model
PhoneLLM Alpha 1 is a voice-agent / phone-conversation model with tool-use and function-calling support, fine-tuned by Pipecat AI from NVIDIA-Nemotron-3-Nano-30B-A3B (a Nemotron-H hybrid Mamba–attention mixture-of-experts model, 30B total / ~3B active parameters).
Conversion details
- Converted with llama.cpp release b10672 (
convert_hf_to_gguf.py→ F16, thenllama-quantize) - Source weights: BF16 safetensors from the official repository (revision
8e76aaa)
Files
| File | Quant | Size |
|---|---|---|
| phonellm-alpha-1-F16.gguf | F16 | 63.2 GB |
| phonellm-alpha-1-Q8_0.gguf | Q8_0 | 33.6 GB |
| phonellm-alpha-1-Q4_K_M.gguf | Q4_K_M | 24.5 GB |
| phonellm-alpha-1-Q3_K_M.gguf | Q3_K_M | 19.8 GB |
⚠️ Important: these quants are larger than their names suggest
This architecture's weight tensors have column counts of 1856, 2688 and 3712 — none of which are divisible by 256, the superblock size that llama.cpp's K-quants require. As a result llama-quantize reports 134 of 401 tensor(s) required fallback quantization and substitutes the legacy block-32 types defined in its fallback table (src/llama-quant.cpp): Q3_K → Q4_0, Q4_K → Q5_0, Q6_K → Q8_0.
Only the 29 tensors with a 4096 column count keep their requested K-quant type. The actual composition by size is:
| File | Real tensor mix |
|---|---|
| Q8_0 | 99.9% Q8_0 |
| Q4_K_M | 66.7% Q5_0, 32.5% Q8_0, 0.7% Q4_K |
| Q3_K_M | 49.4% Q5_0, 45.3% Q4_0, 2.5% Q5_1, 1.9% Q8_0, 0.5% Q3_K |
Practical implications:
- The files are larger than you would expect from these quant names on a conventional dense model. Q3_K_M is ~19.8 GB, not the ~13 GB a 30B Q3_K_M would normally be.
- Quality should be better than the names imply, since the fallbacks are higher-precision than the requested types.
- No Q6_K is published. Its fallback type is Q8_0, so a Q6_K build came out 99.1% Q8_0 tensors at 33.5 GB — within 0.1 GB of the Q8_0 file and marginally worse. Use
Q8_0instead.
Usage
Requires a recent llama.cpp build (b10672 or newer) with Nemotron-H support:
llama-cli -m phonellm-alpha-1-Q4_K_M.gguf -cnv
Or serve an OpenAI-compatible API:
llama-server -m phonellm-alpha-1-Q4_K_M.gguf
License
PhoneLLM Alpha 1 is released under the BSD 2-Clause License (see LICENSE).
PhoneLLM is a derivative work of NVIDIA Nemotron 3 Nano 30B-A3B, which is licensed under the NVIDIA Nemotron Open Model License (see LICENSE_NVIDIA.txt). Per Section 3 (Redistribution) of that license, this repository includes a copy of the NVIDIA Nemotron Open Model License and retains the NVIDIA copyright and attribution notices. The BSD 2-Clause terms apply to Pipecat AI's modifications and to the model as a whole, as Section 3 permits; the NVIDIA license continues to apply to the underlying Nemotron work. "Nemotron" and "NVIDIA" are trademarks of NVIDIA Corporation, used here only to describe the origin of the base model.