Qwen3.6-35B-A3B - GGUF

NVFP4 GGUF quantization of WhiskyAKM/Qwen3.6-35B-A3B-GGUF, originally based on Qwen/Qwen3.6-35B-A3B — a Mixture-of-Experts multimodal model from the Qwen3.6 series with 35B total parameters and only 3B activated per token.

This repository contains an NVFP4 GGUF quantization of the model, making it usable with llama.cpp and other GGUF-compatible inference engines that support the NVFP4 format.

NVFP4 is NVIDIA's 4-bit floating-point quantization format, offering high quality at minimal size. Requires a compatible build of llama.cpp with NVFP4 support.

Model Overview

Qwen3.6-35B-A3B is a multimodal model built by the Qwen Team that handles text, image, and video inputs and generates text output. It uses a hybrid linear attention + full attention architecture with Mixture-of-Experts, achieving strong agentic coding and reasoning performance while keeping the activated parameter count low for efficient inference.

GGUF Files

A chat_template.jinja file is also provided for use with chat-based inference.

Note on mmproj: The mmproj.gguf file contains the vision projector needed for multimodal (image/video) inference. It is shared across all quantization variants.

Usage

llama.cpp (CLI)

# Run text-only inference
./llama-cli \
  -m qwen3.6-35b-a3b-nvfp4.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 1.0 --top-k 20 --top-p 0.95

llama-server (OpenAI-compatible API)

# Text-only
./llama-server \
  -m qwen3.6-35b-a3b-nvfp4.gguf \
  --host 0.0.0.0 --port 8080

# Multimodal (image + video)
./llama-server \
  -m qwen3.6-35b-a3b-nvfp4.gguf \
  --mmproj mmproj.gguf \
  --host 0.0.0.0 --port 8080

Multimodal (Image / Video)

For image and video inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing image/video data alongside text prompts.

Modality order tip: For best results, place image/video content before text in your prompt.

Generation Parameters

Recommended parameters from the model's generation_config.json:

Recommended Sampling by Mode

Thinking Mode

Qwen3.6 operates in thinking mode by default, generating reasoning content signified by <think>\n...\n</think>\n\n before producing the final response.

• Enable (default): The model automatically generates thinking content before its final answer.
• Disable: Set enable_thinking to false in the chat template kwargs (or equivalent in your inference engine) to obtain direct responses without thinking.

Qwen3.6 does not support the /think and /no_think soft switches from Qwen3. Thinking is controlled via API/chat template parameters.

Preserve Thinking

Qwen3.6 can optionally retain and leverage thinking traces from historical messages. Enable preserve_thinking in the chat template kwargs to keep full reasoning context across turns - particularly beneficial for agentic scenarios where it can improve decision consistency and reduce overall token consumption.

Key Features

• Mixture of Experts: 35B total parameters with only 3B activated per token for efficient inference
• Hybrid Attention: Gated DeltaNet (linear attention) with full attention every 4th layer
• Multimodal: Text, image, and video understanding
• Long Context: 262,144 token context window, extensible to 1,010,000 tokens via YaRN
• Agentic Coding: Strong performance on SWE-bench, Terminal-Bench, and repository-level reasoning
• Function Calling: Native support for structured tool use via <function=...> format
• Multi-Token Prediction: Trained with MTP for speculative decoding acceleration
• Multilingual: Support for 140+ languages

Processing Ultra-Long Texts

For contexts exceeding 262,144 tokens, YaRN RoPE scaling is recommended. The following RoPE parameters can be used to extend context up to ~1M tokens:

{
    "mrope_interleaved": true,
    "mrope_section": [11, 11, 10],
    "rope_type": "yarn",
    "rope_theta": 10000000,
    "partial_rotary_factor": 0.25,
    "factor": 4.0,
    "original_max_position_embeddings": 262144
}

Note: Static YaRN applies a constant scaling factor regardless of input length, which may impact performance on shorter texts. Only enable YaRN when long context is required, and adjust factor as needed (e.g., factor: 2.0 for ~524K context).

Acknowledgements

• GGUF source: WhiskyAKM/Qwen3.6-35B-A3B-GGUF
• Original model: Qwen/Qwen3.6-35B-A3B
• Blog post: Qwen3.6-35B-A3B

Citation

@misc{qwen36_35b_a3b,
    title = {{Qwen3.6-35B-A3B}: Agentic Coding Power, Now Open to All},
    url = {https://qwen.ai/blog?id=qwen3.6-35b-a3b},
    author = {{Qwen Team}},
    month = {April},
    year = {2026}
}

License

Apache License 2.0