Qwythos-27B-v1-GGUF

Developed by Empero

GGUF quantizations of empero-ai/Qwythos-27B-v1 for llama.cpp, Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes. Qwythos-27B-v1 is a dense, hybrid Qwen3.5-27B—the larger continuation of Qwythos-9B, trained on the same curriculum at a larger scale. Nothing was removed to make this conversion fit: its native MTP head, vision tower, and 1,048,576-token (1M) context capability are intact. This is the pre-RL v1 checkpoint, following SFT, DPO, and ESFT; an RL-trained v2 will follow, while v1 is a complete checkpoint in its own right.

For the full capability, training, and evaluation writeup, see the base model card. This card is intentionally focused on choosing a file and running it.

Files

The repository is named Qwythos-27B-v1-GGUF, but the files on disk deliberately use the shorter Qwythos-27B-* names below. Use those exact filenames when downloading or passing -m.

Normal text weights — trunk only (64 blocks)

If you do not know which to pick, start with Q4_K_M.

MTP-enabled text weights (nextn_predict_layers = 1)

These include the native MTP head inside the GGUF. Use them with a recent llama.cpp build that supports MTP draft speculation, for example --spec-type draft-mtp.

Vision projector — for image input

The vision tower is intact in this release. Download the mmproj alongside a text quant whenever you want image input.

What fits on a GPU?

These are practical weight-size-based estimates, not measured performance or VRAM benchmarks. They assume a modest context and leave room for runtime overhead and the KV cache; image input adds the 0.86 GiB mmproj. Longer contexts—especially the full 1M window—make the KV cache the dominant memory cost and may require CPU/KV offload or multiple GPUs regardless of weight quant.

Measured: Qwythos-27B-Q4_K_M.gguf fully offloaded to a single RTX 5090 (32 GB) at 4k context runs at ~74 tok/s generation, ~622 tok/s prompt (llama.cpp CUDA build, -ngl 99). One data point on one card, not a benchmark sweep — but it confirms a 27B at Q4_K_M is comfortably interactive on a single consumer GPU.

Quick start

llama.cpp (llama-cli)

llama-cli \
  -m Qwythos-27B-Q4_K_M.gguf \
  -p "Explain the design tradeoffs in a Gated-DeltaNet hybrid model." \
  -n 16384 \
  --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
  -c 16384

llama.cpp (llama-server)

llama-server \
  -m Qwythos-27B-Q4_K_M.gguf \
  -c 16384 --port 8080

Ollama

ollama run hf.co/empero-ai/Qwythos-27B-v1-GGUF:Q4_K_M

LM Studio / jan / KoboldCpp

Download any text .gguf from this repository and load it in your runtime. Preserve the embedded Qwythos chat template if the runtime asks you to select or configure one.

llama.cpp with MTP draft speculation

llama-server \
  -m Qwythos-27B-MTP-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  -c 16384 --port 8080

MTP support requires a recent llama.cpp build. If your runtime does not support MTP draft speculation, choose a trunk-only file instead.

Vision (image input)

Download a text quant and mmproj-Qwythos-27B-F16.gguf, then use llama.cpp's multimodal CLI or server.

llama.cpp (llama-mtmd-cli)

llama-mtmd-cli \
  -m Qwythos-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-27B-F16.gguf \
  --image ./photo.jpg \
  -p "Describe this image in detail." \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  -c 16384

llama.cpp server

llama-server \
  -m Qwythos-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwythos-27B-F16.gguf \
  -c 16384 --port 8080

Hybrid-precision quantization (Gated-DeltaNet / SSM tensors)

Qwythos is a hybrid model: three Gated-DeltaNet linear-attention layers for every full-attention layer. Its Gated-DeltaNet state tensors are disproportionately sensitive to low-bit quantization, so these K-quants retain them at higher precision than the surrounding weights:

The entire Gated-DeltaNet state path is held at Q8_0 or better in every K-quant — no part of it is allowed to drop to Q4/Q5/Q6, regardless of the nominal quant level. This costs roughly 2–4% in file size over a flat K-quant and is the single most important difference between these files and a naive conversion of a hybrid model. Q8_0 and BF16 are uniform and need no overrides. MTP variants additionally pin nextn.eh_proj to Q8_0.

Built with llama.cpp pinned at commit d0f9d2e5, using architecture qwen35.

Identity and chat template

The embedded chat template identifies the model as: “You are Qwythos, a model created by Empero AI. Only bring up your identity if the user asks.” Use the runtime's chat/completions mode rather than hand-rolling a different prompt format where possible.

Function calling

The embedded chat template carries Qwen3.5's full tool-use spec, so tool calling works through llama-server's OpenAI-compatible endpoint without a wrapper — pass tools in your /v1/chat/completions request and parse the emitted <tool_call> blocks:

<tool_call>
<function=NAME>
  <parameter=PARAM>value</parameter>
</function>
</tool_call>

Given the terminal/agentic training behind this checkpoint, tool-augmented use is the intended deployment path.

Sampling recommendations

Qwythos is a reasoning model — responses open with a <think>…</think> block before the final answer. These match the shipped generation_config.json defaults:

As llama.cpp flags:

--temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 -n 16384

Use repeat_penalty 1.05 rather than Qwen's default of 1.0 — it prevents rare non-terminating reasoning loops on long generations.

Long context (1M tokens)

The model configuration supports a 1,048,576-token window: a 4× YaRN extension over the 262,144-token native window. Set the runtime context to the amount you actually need—lower -c values substantially reduce KV-cache memory requirements.

The 1M setting is an inference-time capability, not a promise that a single GPU can hold a 1M KV cache. For roughly 512k context, a YaRN factor of 2.0 is preferable; if you will never exceed the native 262k window, remove scaling entirely. Static YaRN scaling can slightly degrade short-context quality. For the full 1M window, plan for multi-GPU or aggressive KV offload.

Verification

SHA256SUMS is included in this repository. Verify a download before loading it:

sha256sum -c SHA256SUMS

Limitations

• Pre-RL v1. This release has completed SFT, DPO, and ESFT, but the planned RL-trained v2 is still to come.
• Context costs memory. Weight size is only part of the hardware budget; long contexts need substantial KV-cache memory.
• MTP is runtime-dependent. Use the trunk-only files when your GGUF runtime does not yet support draft-mtp speculation.
• No standardized academic benchmark suite has been run for this checkpoint. Validate the model on the tasks and operating conditions that matter to your deployment.
• Uncensored model. Add the application-level review and safety controls appropriate to an end-user deployment.

Stay in the loop

Sign up for the Empero newsletter at empero.org for releases, evals, and research notes.

Support / Donate

If this model helped you, consider supporting the project:

• BTC: bc1qx6zepu6sfkvshgdmc4ewu6pk6rpadvpgffpp7v
• LTC: ltc1qv2mefzps2vtjcpwfx8xxdrpplrcvltswm68r7x
• XMR: 42Dbm5xg5Nq26fdyzfEU7KBnAJfhi7Cvz5J2ex5CzHXkfKuNEJzYCcmJ1GTbgjFZ5MBx72sdG1G9239Cd6rsZfv4QeDkYJY

Provenance & licensing

Qwythos-27B-v1 is derived from Qwen3.5-27B. The weights are released under Apache-2.0, inherited from the Qwen3.5-27B base, and are shared for research and experimentation as-is.

Acknowledgements

• Developed and released by Empero
• Base model: Qwen3.5-27B (Alibaba Qwen team)
• Training: TRL + Transformers
• GGUF quantization: llama.cpp (ggml-org)