PXA PXQ GGUF quantizations of Ornith-1.5-9B

PXA/PXQ GGUF quantizations of ornith-ai/Ornith-1.5-9B.

These files use the standard GGUF container, but contain PXA-specific PXQ tensor types. Stock upstream llama.cpp builds that do not implement the PXA PXQ types are not expected to load these files.

Runtime support

• PXA — https://github.com/poisonxa16/pxa — the reference PXQ runtime. Its primary optimized targets are NVIDIA Pascal (SM60/SM61) and Volta (SM70). PXA also documents wide CUDA builds including Ampere (SM86) and Ada (SM89); those newer architectures use safe generic paths where a Pascal/Volta-specific kernel does not apply and are not claimed to have the same tuning coverage.
• mistrjirka/llama.cpp v100-optimized — https://github.com/mistrjirka/llama.cpp/tree/v100-optimized — a current llama.cpp-based runtime focused on NVIDIA Volta (SM70) and Turing (SM75), tested primarily on Tesla V100 and RTX 2080 Ti. The branch implements the complete published PXQ tensor family, not just PXQ4.

Both runtimes also understand mixed files containing multiple PXQ tensor tiers. Performance and fast-path coverage still depend on GPU architecture and tensor shape; "supported" here means the PXQ wire type and execution path exist, not that every tier is equally optimized on every GPU.

Model details

• Original model: ornith-ai/Ornith-1.5-9B
• Parameter count: ~9B, dense
• Architecture in GGUF metadata: qwen35
• Native context length: 262,144 tokens
• Input: text; the original model also supports image input with a compatible mmproj (projector not included in this repo)
• Speculative decoding: no MTP head in this release
• Importance matrix: not used for these PXA quantizations

Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Which quant should I choose?

Start with PXQ4. It is the normal PXA 4-bit core tier and is the best default here for size / fidelity / PXA decode support.

• Choose PXQ6 when fidelity matters more than size and decode efficiency.
• Choose PXQ4-HQ for a smaller quality step above PXQ4 using finer bs8 sub-scales.
• Choose PXQ3 or PXQ2 when VRAM/RAM pressure is more important than quality.
• PXQ1, if present, is the extreme-compression stretch tier and should be treated as experimental.

The bpw values below describe the PXQ codec used for the main quantized tensors, not the effective bpw of the complete file. PXA intentionally preserves sensitive/structurally unsuitable tensors in Q6_K, Q8_0, F16/F32 or other higher-precision types.

Available files

All published model filenames explicitly contain GGUF.

PXQ-UNIVERSAL / PXQU

PXQ_UNIVERSAL is not a single fixed quantization tier. In PXA it requires an explicit per-tensor .tiers map (--pxq-universal) and the quantizer intentionally refuses to create a PXQU file without one. This release therefore does not invent a generic PXQU artifact for Ornith. A future PXQU file should use an Ornith-specific map and be labeled with that map/preset.

Download

Install the Hugging Face CLI, then download a specific quant, for example PXQ4:

hf download mistrjirka/Ornith-1.5-9B-PXQ-GGUF \
  --include 'Ornith-1.5-9B-PXQ4-GGUF.gguf' \
  --local-dir ./

Checksums are in SHA256SUMS and alongside each uploaded GGUF as .sha256 files.

Running with PXA

Use the PXA fork/runtime rather than a stock llama.cpp binary:

git clone https://github.com/poisonxa16/pxa.git
cd pxa
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-cli

Then run a downloaded file, for example:

./build/bin/llama-server \
  -m Ornith-1.5-9B-PXQ4-GGUF.gguf \
  -c 262144 \
  -ngl 99 \
  --jinja

Tune context/KV types and GPU offload for your hardware. The model itself supports a 262k native context, but actually allocating that full context can require substantial memory.

Quantization implementation

The files were quantized CPU-only on a SpacemiT X60 RISC-V host (Orange Pi RV2-class) with an RVV/vector-FP16 build:

-march=rv64gcv_zba_zbb_zbc_zbs_zfh_zvfh -mabi=lp64d

PXA base revision:

6d7dbfff653288e5f83fad5f7ee443329b77ebfc
PXA v2026.09.07-rc1

A local RISC-V quantization patch (25d34ee) added a minimal CPU-only quantizer frontend, portable FP16 references, byte-based progress reporting, and a bounded-memory BF16/F16 → Q6_K/Q8_0 row path. The bounded-memory Q6_K/Q8_0 path was verified byte-identical to whole-tensor quantization.

The RISC-V build was also checked to contain real RVV instructions. A clean PXQ3 reference A/B on this X60 measured ~1.9× faster with RVV than the same source compiled without RVV, with byte-identical output.

Validation

Each full quantization is accepted only after PXA's end-of-run composition checks. The original PXQ4 artifact was additionally reopened with PXA's GGUF reader and parsed as all 427 tensors successfully.

Original model

This repository contains quantizations only. Model authorship, intended use, limitations and license come from the original Ornith-1.5-9B release. The original model is MIT licensed.