⚡ FlashModel-Qwen3.5-9B — GGUF

imatrix GGUF quants of FlashModel-Qwen3.5-9B, a reasoning / coding / tool-calling fine-tune of Qwen/Qwen3.5-9B distilled from open-weight frontier teachers.

• All quants use an importance matrix computed on this fine-tune (not borrowed from the base model).
• *_K_XL quants use a dynamic per-tensor layout mirrored tensor-for-tensor from unsloth/Qwen3.5-9B-GGUF (see How these were made).
• Every file was loaded and ran a generation test in llama.cpp before upload.
• Text-only: the fine-tune does not include the vision tower, so there is no mmproj.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Which one should I pick?

Leave room for the KV cache: long reasoning traces need context. Size numbers are file sizes, not total memory use.

Usage

llama.cpp

# download one quant
huggingface-cli download flashback2k/FlashModel-Qwen3.5-9B-GGUF \
  FlashModel-Qwen3.5-9B-Q4_K_XL.gguf --local-dir .

# chat (thinking mode, recommended sampling from Qwen3.5)
llama-cli -m FlashModel-Qwen3.5-9B-Q4_K_XL.gguf --jinja -ngl 99 -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5

# OpenAI-compatible server with tool calling
llama-server -m FlashModel-Qwen3.5-9B-Q4_K_XL.gguf --jinja -ngl 99 -c 32768 --port 8080

Or pull straight from the Hub: llama-server -hf flashback2k/FlashModel-Qwen3.5-9B-GGUF:Q4_K_XL --jinja

Ollama

ollama run hf.co/flashback2k/FlashModel-Qwen3.5-9B-GGUF:Q4_K_XL

LM Studio

Search for flashback2k/FlashModel-Qwen3.5-9B-GGUF and pick a quant.

Recommended sampling

These are the base model's official recommendations (Qwen3.5 model card); they were not re-tuned for the fine-tune.

Always pass --jinja so the embedded Qwen3.5 chat template (thinking blocks, tool calls) is used.

Reasoning-budget tag

Training system prompts started with a reasoning-budget tag chosen from the length of the teacher's reasoning:

<|reasoning_budget|>medium<|/reasoning_budget|>

Values: off, low, medium, high, xhigh, max. The tag's effect on output length has not been measured yet; treat it as experimental.

About the fine-tune

Training data and teachers

Only open-weight teachers whose licenses allow training on their outputs:

Datasets © NVIDIA, CC BY 4.0 (some Math StackExchange-derived samples CC BY-SA 4.0).

⚠️ Status and known limitations

• No benchmarks yet. Lower held-out loss means the model imitates the teachers more closely; it does not by itself prove it beats the stock Qwen3.5-9B. Comparative evals are planned and will be added here.
• English-centric. All training data was English. On non-English prompts (e.g. Russian) the model often reasons in English.
• Math and code examples longer than 16k tokens were excluded, which skews those domains toward shorter problems.
• Inherits the base model's limitations and biases.

How these were made

1. Merged weights → BF16 GGUF with llama.cpp convert_hf_to_gguf.py (build b11005-82324fc50).
2. imatrix: llama-imatrix on the BF16 model over ~131k tokens (256 × 512) of held-out training-distribution text (math, code, tool calls, instructions, science). A Q8_0 source produced non-finite activations — the stock Qwen3.5-9B Q8_0 does too — so BF16 was used.
3. Standard quants: llama-quantize --imatrix.
4. *_K_XL quants: Unsloth has not published its Dynamic quantization tool, but the resulting per-tensor types are visible in its GGUF headers. The types of all 427 tensors were read from unsloth/Qwen3.5-9B-GGUF (UD-Q3_K_XL, UD-Q4_K_XL, UD-Q5_K_XL), applied with llama-quantize --tensor-type-file, and verified tensor-for-tensor after quantization. The layout keeps the most sensitive tensors at higher precision — e.g. in Q4_K_XL: Gated DeltaNet ssm_out at Q8_0, ssm_alpha/ssm_beta at F16, attn_v at Q6_K, about half of ffn_down at Q6_K. The imatrix is this model's own. These are not official Unsloth quants and are not affiliated with or endorsed by Unsloth.
5. Each quant loaded in llama.cpp (CUDA) and generated a reply before upload.

Credits

Qwen team for Qwen3.5 · NVIDIA for the Nemotron SFT datasets · DeepSeek, Zhipu AI (GLM), OpenAI (GPT-OSS) and Moonshot AI (Kimi) for open-weight teachers · ggml-org/llama.cpp · Unsloth for the published dynamic-quant layouts · quant-table format inspired by mradermacher.