Ornith 1.5 Cyclos 1M Uncensored — GGUF (Q4_K_M)

GGUF build of a LoRA fine-tune of 0xKitkat/Ornith-1.5-35B-A3B-Uncensored, itself an uncensored derivative of ornith-ai/Ornith-1.5-35B-A3B. Extended to 1,048,576 context via YaRN, with a companion vision projector.

Everything this model is good at comes from Ornith-1.5 — its coding and agentic post-training, its self-improvement RL, the native vision tower, the 262K context window and the multi-token-prediction head are all upstream work by the Ornith Team.

All credit for the underlying model belongs upstream — see Credits. This repository contributes only a fine-tune, a context-window change, and a quantization.

Files

Both GGUF files are required for image/video input. Most runtimes pair them automatically from the mmproj- prefix when they sit in the same directory. The config.json is there for vLLM's GGUF plugin — see Runtime support for why that still is not enough to load this model in vLLM.

Changes from the base model

Stated per Apache-2.0 §4(b):

1. LoRA fine-tune (SFT), merged into the base weights. r=16, alpha=16, use_rslora=true (so scaling is alpha/sqrt(r) = 4.0). Targets attention and shared-expert projections plus, via PEFT target_parameters, the fused MoE expert tensors mlp.experts.gate_up_proj and mlp.experts.down_proj. Training focus: coding, market/trading analysis, content classification, and tool/function calling.
2. Context extended 262,144 -> 1,048,576 via YaRN (factor 4.0, original_context_length 262144), written into the GGUF metadata.
3. Quantized to Q4_K_M; vision projector exported separately at F16.

The vision tower was not fine-tuned — every adapter tensor lives under language_model.*, and model.visual.* is bit-identical to the base after merging. Image and video ability is inherited from upstream, neither improved nor degraded.

Honest limitations

• The 1M context is extrapolation, not training. The base is natively 262,144 with no RoPE scaling; YaRN extends it at inference time. Quality degrades as you go further beyond 262,144, and this has not been measured. Treat 1M as an upper bound, not a validated working length.
• No audio. This architecture has no audio encoder. <|audio_start|>-style tokens exist in the tokenizer but are vestigial shared-vocabulary entries with nothing behind them; runtimes report audio: false. Audio needs an external ASR front-end.
• No benchmarks are claimed. No evaluation suite was run against this build. Upstream benchmark numbers do not transfer to it.
• "Uncensored" is inherited from the base, which reduced refusal behaviour. It does not mean every request is answered, and it does not remove your responsibility for how the model is used.
• Q4_K_M is a lossy quantization; at 1M context, q4_0 KV cache compounds that. Small reasoning budgets measurably degraded arithmetic accuracy in local testing.

Runtime support (tested)

Tested locally on 2 x NVIDIA RTX 5060 Ti (16 GB each, 31.7 GB total), CUDA 13.1, 59 GB system RAM.

vLLM + GGUF does not work — and why

A config.json is included in this repo, because vllm_gguf_plugin reads the config from the GGUF's own directory. That is necessary but not sufficient. With it in place the loader gets past config parsing and fails at weight mapping:

RuntimeError: Failed to map GGUF parameters (550):
  ['model.layers.0.linear_attn.dt_bias', 'model.layers.0.linear_attn.A_log', ...]

Qwen3.5-MoE is a hybrid: 30 of its 40 layers are linear (SSM-style) attention, and the plugin's generic weights adapter has no mapping for those GGUF tensor names. Fixing it needs a per-architecture adapter in vllm_gguf_plugin/weights_adapter/, which does not exist for this model. Nothing in this repository can work around it.

The config is flattened (model_type: qwen3_5_moe, architectures: ["Qwen3_5MoeForCausalLM"]) rather than the nested multimodal form, because the plugin calls AutoModelForCausalLM.from_config and the composite config raises AttributeError: 'Qwen3_5MoeConfig' object has no attribute 'vocab_size'. It is kept in case a future plugin release adds the missing adapter.

vLLM + BF16 works

Verified end-to-end with coherent generation:

vllm serve <bf16-repo> \
  --tensor-parallel-size 2 --max-model-len 4096 \
  --gpu-memory-utilization 0.92 --cpu-offload-gb 22 --enforce-eager

BF16 weights are ~67 GB against 31.7 GB of VRAM, so roughly 44 GB is offloaded to system RAM. That works but is offload-bound: about 1.8 tok/s on this machine, versus 19-57 tok/s for the Q4_K_M GGUF under llama.cpp. On this hardware the GGUF is the practical choice; BF16 in vLLM is worth it only with more VRAM, where it also unlocks the MTP head (Qwen3_5MoeMTP) for speculative decoding — a head llama.cpp discards, which is what the harmless blk.40.* unused tensor warnings refer to.

Long context in vLLM/SGLang comes from config.json: rope_parameters with rope_type: "yarn", factor: 4.0, original_max_position_embeddings: 262144. vLLM 0.27.1 reads the transformers-5 rope_parameters key. Raise --max-model-len toward 1048576 only with the KV memory to match.

The 1M context, in detail

context_length in this GGUF is 1,048,576, reached by YaRN scaling (factor 4.0, original_context_length 262144) baked into the file metadata. No flags needed — llama.cpp reads it and reports n_ctx_slot = 1048576 on load.

What it is not: trained. The base model is natively 262,144 with rope_type: default and no scaling block. YaRN is an inference-time frequency interpolation, applied here after the fact. Quality past 262,144 is unmeasured — no long-context evaluation was run. Treat 1M as a mechanical upper bound, not a validated working length. If you need confidence rather than reach, run at the native length instead:

llama-server -m ornith-1.5-cyclos-1M-uncensored-q4.gguf -c 262144   # native, no extrapolation

Why 1M is even feasible here. Qwen3.5-MoE is hybrid: only 10 of its 40 layers use full attention (every 4th), the rest are linear/SSM layers whose state is constant-size rather than growing per token. So the KV cache is roughly a quarter of what a dense 35B would need — about 20 KiB per token at f16.

Measured memory, on 2 x RTX 5060 Ti (31.7 GB VRAM total):

At full 1M with f16 KV the process needed 41.5 GiB RSS; with -ctk q4_0 -ctv q4_0 the same context fits entirely in VRAM. Two practical consequences:

• Quantize the KV cache for anything near 1M. It is the difference between fitting and not.
• Use --fit on, not -ngl 99. Forcing full offload allocated weights and KV successfully and then failed on compute buffers (failed to allocate CUDA0 buffer of size 4182280960). --fit places layers to match available memory.

One accuracy note: q4_0 KV plus a small reasoning budget measurably degraded arithmetic in local testing (a compounding-percentage question came back 5.71% instead of 5.678%). For numeric work, keep --reasoning-budget -1 and consider q8_0 KV at a shorter context.

Audio via a Whisper front-end

This model cannot hear. It has no audio encoder — see Honest limitations. What works is transcribing first and passing the text in, which composes fine with the native image/video path (audio transcript and an image in the same request both land in the prompt).

For convenience this repo ships whisper-ggml-large-v3-turbo.bin — an unmodified copy of OpenAI's Whisper large-v3-turbo as distributed by ggml-org. It is not fine-tuned for this model and is not covered by this repo's Apache-2.0 chain; it carries Whisper's own license and provenance. You can equally fetch it yourself:

cd whisper.cpp && bash models/download-ggml-model.sh large-v3-turbo

audio_chat.py (also included) wires the two together — it converts any ffmpeg-readable file to 16 kHz mono, transcribes with whisper.cpp, and sends the transcript to a running llama-server:

python audio_chat.py meeting.m4a "Summarise the action items."
python audio_chat.py clip.wav --transcribe-only

It needs no editing — the whisper model defaults to the .bin sitting beside it in this repo. Override with environment variables if your layout differs:

WHISPER_CLI=/path/to/whisper.cpp/build/bin/whisper-cli \
WHISPER_MODEL=/path/to/ggml-large-v3-turbo.bin \
ORNITH_URL=http://127.0.0.1:8100 \
  python audio_chat.py meeting.m4a "Summarise the action items."

Build whisper.cpp from https://github.com/ggml-org/whisper.cpp (cmake -B build -DGGML_CUDA=ON && cmake --build build -j --config Release).

Run whisper on CPU (-ng, the script's default) while the LLM is loaded. The model occupies ~29 GB of a 32 GB dual-GPU rig; whisper then cannot allocate and segfaults inside whisper_model_load rather than reporting OOM, which is easy to misread as a corrupt model file. On CPU it runs at roughly 1x realtime with 16 threads.

Transcription loses everything non-verbal — tone, emotion, speaker identity, music, background sound. Fine for meetings and dictation; a real limitation for audio classification.

Built with soup-cli

This build was produced with soup-cli — the LoRA merge, the YaRN context config, and the GGUF conversion/quantization all ran through it.

# how this GGUF was produced (after merging the adapter into the base)
soup export -m ./ornith-merged -f gguf -q q4_k_m \
  --llama-cpp /path/to/llama.cpp \
  -o ornith-1.5-cyclos-1M-uncensored-q4.gguf

The 1M YaRN block was generated by soup's own helper rather than hand-written, so the values match what soup would produce for any long-context export:

from soup_cli.utils.long_context import get_rope_scaling_config
get_rope_scaling_config("yarn", 1_048_576, 262_144)
# {'type': 'yarn', 'factor': 4.0, 'original_max_position_embeddings': 262144}

soup can also drive inference against the BF16 weights (it has no llama.cpp/GGUF serving backend — its backends are transformers, vllm, sglang, mii):

soup serve  -m <bf16-repo> --backend vllm --tp 2 --max-model-len 262144
soup infer  -m <bf16-repo> -i prompts.jsonl -o out.jsonl
soup chat   -m <bf16-repo>

Usage (llama.cpp)

Both commands below were verified on 2 x RTX 5060 Ti (16 GB each, 31.7 GB total). Run them from the directory holding the two .gguf files.

Full 1M context, 4 concurrent users

llama-server \
  -m ornith-1.5-cyclos-1M-uncensored-q4.gguf \
  --mmproj mmproj-ornith-1.5-cyclos-1M-uncensored-f16.gguf --no-mmproj-offload \
  -c 1048576 --parallel 4 --kv-unified --fit on \
  -ctk q4_0 -ctv q4_0 -fa on -b 256 -ub 64 \
  --jinja --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --host 127.0.0.1 --port 8100

Measured: 29.3 GB VRAM, 4 users at 157 tok/s aggregate (48 tok/s each; 88 tok/s single-user), and a 312,052-token needle-in-a-haystack prompt answered correctly.

262K context — ~4x faster prompt ingest

llama-server \
  -m ornith-1.5-cyclos-1M-uncensored-q4.gguf \
  --mmproj mmproj-ornith-1.5-cyclos-1M-uncensored-f16.gguf --no-mmproj-offload \
  -c 262144 --parallel 4 --kv-unified --fit on \
  -ctk q4_0 -ctv q4_0 -fa on -b 512 -ub 128 \
  --jinja --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --host 127.0.0.1 --port 8100

Prefill is the tradeoff, not generation: at -ub 64 a 312K-token prompt took 760 s (~410 tok/s ingest). -ub 128 is roughly 4x faster but will not fit the full 1M.

Why each flag is there

Every one of these was the difference between working and failing on a 32 GB rig:

• --kv-unified — do not drop it. Without it llama.cpp divides the context across slots, so -c 262144 --parallel 4 gives each user only 65,536 tokens. With it there is one shared pool and any slot can address all of it (/slots will show n_ctx = the full value on every slot). Use --kv-unified-per-slot N instead if you want a guaranteed floor per user rather than a shared pool.
• -b 256 -ub 64 at 1M. This is what makes 1M load at all. With default batch sizes the compute buffer alone asks for ~3.1 GiB and allocation fails after weights and KV are already placed.
• --no-mmproj-offload. Keeps the ~857 MiB vision projector off the GPU. With it on the GPU, allocation fails by almost exactly that amount. Costs ~10 s per image on CPU.
• -ctk q4_0 -ctv q4_0. 1M of KV is ~21 GiB at f16 versus ~6 GiB here. Not optional at this depth. Only 10 of 40 layers carry a KV cache (the rest are linear/SSM), which is why 1M is reachable at all.
• --fit on, never -ngl 99. Forcing full offload allocates weights and KV successfully and then dies on compute buffers.
• --reasoning-budget -1. Truncating the reasoning trace measurably degraded arithmetic in testing (a compounding-percentage question returned 5.71% instead of the correct 5.678%).

Reasoning and multimodal notes

• This is a reasoning model and emits <think>...</think>. --reasoning-format deepseek puts it in message.reasoning_content instead of content.
• If you send an image or video with a small max_tokens, content can come back empty — the budget was spent on reasoning. Raise max_tokens, or pass chat_template_kwargs: {"enable_thinking": false}. This looks like a failure but is not.
• Send video as a data:video/mp4;base64,... string in an image_url field; there is no video_url in the server API, and the loader falls back to video decoding via system ffmpeg.
• Per-request reasoning_budget_tokens works. reasoning_effort is accepted but ignored by this chat template — it has no reasoning_effort variable.
• --host 127.0.0.1 keeps the server local. 0.0.0.0 exposes an unauthenticated endpoint; add --api-key if you do that.
• A recent llama.cpp is required — the architecture is qwen35moe.

Upstream recommended sampling: temperature=0.6, top_p=0.95, top_k=20.

Credits and attribution

This is a derivative several steps down a chain. Credit belongs to, in order:

• 0xKitkat — Ornith-1.5-35B-A3B-Uncensored, the direct base of this build (Apache-2.0).
• Ornith Team — ornith-ai/Ornith-1.5-35B-A3B, the coding/agentic post-training, vision tower, 262K context and MTP head (card declares MIT).
• Qwen Team — Qwen/Qwen3.6-35B-A3B (Apache-2.0).
• wangzhang / Abliterix — Qwen3.6-35B-A3B-abliterated, the abliterated donor and documented method.
• ggml-org — llama.cpp, whisper.cpp, the GGUF format, and the conversion/quantization tooling.
• OpenAI — Whisper large-v3-turbo, redistributed here unmodified for the optional ASR front-end under its own license; it is not a component of this model.

Licensed under Apache-2.0, inherited from the base model. Upstream components carry their own licenses as listed above.