Ornith-1.5-35B-A3B-TQ3_4S

Ornith-1.5-35B-A3B-TQ3_4S is a compact TurboQuant GGUF build of ornith-ai/Ornith-1.5-35B-A3B.

Required Runtime

This model uses the custom TQ3_4S tensor type and requires turbo-tan/llama.cpp-tq3.

Stock llama.cpp builds without TurboQuant support cannot load this model.

This is the standard 35B-A3B MoE model, not an MTP release. Do not add draft-MTP speculative-decoding flags.

Ornith-1.5-35B-A3B is a Qwen3.5-based multimodal model. Vision is optional — load the mmproj projector to enable image input (see Vision (Multimodal)).

Files

• Ornith-1.5-35B-A3B-TQ3_4S.gguf - main model, 17.00 GiB
• thumbnail.png - model card banner

Architecture

Build the Required Runtime

git clone https://github.com/turbo-tan/llama.cpp-tq3
cd llama.cpp-tq3

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=OFF \
  -DGGML_CUDA_GRAPHS=ON

cmake --build build --target llama-server -j

For an RTX 3090, -DCMAKE_CUDA_ARCHITECTURES=86 may be added explicitly. Use the architecture matching your GPU on other systems.

Recommended Runtime

Validated on an NVIDIA GeForce RTX 3090 Founders Edition with 24 GiB VRAM:

./build/bin/llama-server \
  -m Ornith-1.5-35B-A3B-TQ3_4S.gguf \
  --alias Ornith-1.5-35B-A3B-TQ3_4S \
  --host 127.0.0.1 --port 8080 \
  -c 32768 -np 1 -ngl 99 -fa on \
  -ctk q8_0 -ctv tq3_0 \
  --reasoning off --jinja

Runtime notes:

• -fa on enables flash attention at runtime.
• The validated CUDA build uses GGML_CUDA_FA_ALL_QUANTS=OFF.
• -ngl 99 fully offloads the model on supported GPUs. Avoid partial offload when comparing the published speed.
• Reduce context from 32768 if the available VRAM is lower than 24 GiB.

Quick Smoke Test

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Ornith-1.5-35B-A3B-TQ3_4S","messages":[{"role":"user","content":"Write ONLY the word ok."}],"max_tokens":16,"temperature":0.0}'

Expected assistant content:

ok

Local Measurements

Measured on RTX 3090 FE with the launch settings above. Generation speed is token throughput excluding prompt processing.

Context-Length Decode Speed

Decode throughput measured with 512-token generation across varying prompt lengths. Compared against Ornith-1.0-35B-TQ3_4S (dense 35B, 13 GiB) on the same hardware and binary.

Runtime: turbo-tan/llama.cpp-tq3 commit 47635d703, -ngl 99 -fa on -ctk q8_0 -ctv tq3_0 -c 262144 --reasoning off, NVIDIA RTX 3090 24GB.

The MoE architecture (3B active parameters) delivers a consistent 4–8% decode speedup over the dense 35B variant across all context sizes.

Benchmark Summary

Notes:

• All benchmarks run with --reasoning off for maximum speed
• MoE architecture (3B active parameters) enables fast inference
• Coding and tool calling scores are competitive with dense 35B models
• Instruction following shows room for improvement on complex multi-constraint tasks

Conversion Recipe

# 1. Convert safetensors to F16 GGUF
python convert_hf_to_gguf.py \
  ornith-ai/Ornith-1.5-35B-A3B \
  --outfile Ornith-1.5-35B-A3B-F16.gguf \
  --outtype f16

# 2. Quantize to TQ3_4S
./build/bin/llama-quantize \
  Ornith-1.5-35B-A3B-F16.gguf \
  Ornith-1.5-35B-A3B-TQ3_4S.gguf \
  tq3_4s

Vision (Multimodal)

Ornith-1.5-35B-A3B is a multimodal model. To enable image input, add the vision projector with --mmproj:

./build/bin/llama-server \
  -m Ornith-1.5-35B-A3B-TQ3_4S.gguf \
  --mmproj mmproj-f16.gguf \
  --alias Ornith-1.5-35B-A3B-TQ3_4S \
  --host 127.0.0.1 --port 8080 \
  -c 32768 -np 1 -ngl 99 -fa on \
  -ctk q8_0 -ctv tq3_0 \
  --reasoning off --jinja

Validation Notes

• Benchmark results are local measurements, not claims from the parent model repository.

License

Use is subject to the base model license and the licenses of turbo-tan/llama.cpp-tq3.