Qwen3.5-4B-Prime — GGUF

GGUF quantizations of a Qwen3.5-4B fine-tune trained to drive Emacs through python-bridge/MCP instead of generic write/edit file tools.

Provenance:

Qwen/Qwen3.5-4B  →  empero-ai/Qwen3.8-4B-Distill  →  this fine-tune  →  GGUF (here)

Fine-tuned on Jadson/ox-alpha-pi-traces-emacs-bridge, which rewrites the agent traces of TeichAI/Ox-Alpha-Pi-Traces so every file creation and edit routes through ox-bridge-write, ox-bridge-edit and ox-bridge-read.

Text-only. The base is an image-text-to-text checkpoint, but this one carries no vision tower and ships no mmproj. Images are not supported.

Architecture

Qwen3.5 is a hybrid: 32 layers where 24 use linear attention (gated deltanet) and 8 use full attention, interleaved every 4th layer. 4.21 B parameters, vocab 248320, tied embeddings, 262144 max positions.

Because the embeddings are tied and the vocabulary is large, token_embd alone is 15% of all weights — which is why every recipe below keeps it at Q4_K instead of letting it fall to the 2-bit default.

Files

About the sub-2-bit files

They are published for completeness and reproducibility, not because they work.

Unsloth's UD-IQ1 quants earn their reputation on MoE models of 100 B+ parameters, where expert redundancy absorbs the damage. This is a dense 4 B model: 3.7 B of non-embedding weights with nowhere to hide. Expect heavy degradation at IQ1_M and IQ1_S.

TQ1_0 is stricter still. It is ternarization — designed for models trained ternary, BitNet-style. Applied to ordinary BF16 weights it rounds them to {−1, 0, +1}. The file loads and generates tokens; the output is not worth reading. Q1_0 at 1.125 bpw is in the same territory.

If you want a small usable file, take IQ4_NL.

Quantization recipe

Importance matrix computed with llama-imatrix over 319 chunks at n_ctx=512, directly from the BF16, covering all 248 quantizable trunk tensors (24 linear-attention layers × 5, 8 full-attention layers × 4, 32 layers × 3 FFN).

Two calibration sources, and the second one is the point:

• calibration_datav3.txt — generic mixed-register English, for breadth.
• The training distribution itself. 110 sessions sampled from ox-alpha-pi-traces-emacs-bridge (the ipython split, matching the format the model actually emits), rendered through the chat template with <think> blocks and ```ipython cells, and tokenized with --parse-special so <|im_start|> lands as the real control token rather than literal text.

Calibrating on the deployment distribution is not cosmetic. Against a generic-only matrix, 65 of 248 tensors shifted their importance profile by more than 10% (cosine similarity < 0.90; mean 0.9285, minimum 0.4187), and the largest shifts land on ffn_down and ssm_out — precisely the projections the recipes below protect, and the ones that decide quality at low bit-width. Importance also spread out slightly (top-decile share 0.482 → 0.466), as more diverse calibration should.

One caveat for anyone reproducing this: part of that shift comes from simply having more data (70 → 319 chunks reduces noise), not only from the domain match. The two effects are not separated here.

UD-* is not a llama-quantize type — it is Unsloth's naming for a dynamic mixture, per-tensor rather than uniform. Reproduced here with --tensor-type overrides (first matching regex wins and disables the automatic mixture for that tensor):

llama-quantize --imatrix imatrix.gguf \
  --tensor-type "ssm_alpha=f16" \
  --tensor-type "ssm_beta=f16" \
  --tensor-type "ssm_conv1d=f16" \
  --token-embedding-type q4_k --output-tensor-type q6_k \
  --tensor-type "blk\.(0|1|31|32)\.=q4_k" \
  --tensor-type "attn_k=q4_k" --tensor-type "attn_v=q4_k" \
  --tensor-type "attn_output=iq2_xxs" --tensor-type "ssm_out=iq2_xxs" \
  --tensor-type "ffn_down=iq2_xxs" \
  model-BF16.gguf model-UD-IQ1_S.gguf IQ1_S 20

What is protected and why:

IQ4_NL uses the stock mixture plus the three ssm_* protections.

Multi-token prediction (MTP)

All files carry an MTP block at blk.32 for speculative decoding:

llama-cli -m Qwen3.5-4B-Prime-IQ4_NL.gguf --spec-type draft-mtp -p "..."

Measured on this machine (CPU, 4 threads, IQ4_NL): 6.8 → 10.0 tok/s, a 47% speedup, with no fallback warnings.

Disclosure: the fine-tuned checkpoint shipped without an MTP head, so these 15 tensors were grafted from Qwen/Qwen3.5-4B base. The head therefore predicts against base weights, not the fine-tune. This lowers the acceptance rate; it does not affect correctness, because the target model verifies every drafted token. It also receives no imatrix calibration — a plain forward pass never exercises the MTP graph — which is why blk.32 is pinned to Q4_K.

At 1-bit the MTP head is pointless regardless.

System prompt — required

Use SYSTEM_PROMPT.md. It ships in this repo and it is not optional: without it the model still emits ipython cells, but falls back to ! bash python3 -c "..." instead of the Emacs bridge.

Same prompt, same seed, IQ4_NL — without the system prompt:

result = ! bash python3 -c "
with open('hello.py', 'w') as f:
    f.write('''def greet(name='World'): ...''')
"

With it:

result = await mcp.call_tool("tools", "ox-bridge-write", {"path": "hello.py", "content": "..."})
print(result)

Usage

# chat
llama-cli -m Qwen3.5-4B-Prime-IQ4_NL.gguf -c 8192 -sysf SYSTEM_PROMPT.md

# server, with speculative decoding
llama-server -m Qwen3.5-4B-Prime-IQ4_NL.gguf -c 8192 --spec-type draft-mtp

Requires a llama.cpp build with the qwen35 architecture (src/models/qwen35.cpp). Older builds will refuse to load these files.

The model is a reasoning model and opens a <think> block by default. Suppress it with enable_thinking=false in the chat template.

Reproducing

python convert_hf_to_gguf.py ./model-dir --outtype bf16 --outfile BF16.gguf
llama-imatrix -m BF16.gguf -f calib_domain.txt -o imatrix.gguf \
              -c 512 -b 2048 -ub 512 -np 4 -t 20 -ngl 0 --no-ppl --parse-special
# then the llama-quantize invocations above

On a memory-constrained machine add --max-buffer-size 256 to every llama-quantize call — the default caps tensor rows held in RAM at 8192 MiB, which a 248320×2560 embedding will happily exceed.

License and credits

Apache-2.0, inherited from Qwen3.5-4B. The training dataset is MIT.

• Qwen/Qwen3.5-4B — base model
• empero-ai/Qwen3.8-4B-Distill — distilled intermediate
• Jadson/ox-alpha-pi-traces-emacs-bridge — training dataset
• TeichAI/Ox-Alpha-Pi-Traces — source traces the dataset is derived from
• Dynamic per-tensor quantization method popularized by Unsloth