Qwen3.6-35B-A3B-Heretic — REAP-pruned (30%, 256→180 experts)

Overview

Qwen3.6-35B-A3B-Heretic-REAP-0.30 is a model checkpoint packaged for compatible Hugging Face runtimes, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

The repository name identifies a behavior-modified or reduced-filtering lineage. That label describes the source or conversion history; it is not a guarantee of unrestricted behavior in every prompt or runtime. Test outputs carefully before sharing or deploying them.

At a glance

What is included

• *.safetensors (3 files)
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja
• Additional configuration, tokenizer, processor, or shard files (9 visible artifacts total)

Quick start

Getting started

Start with the upstream library named in the repository metadata and keep all configuration, tokenizer, processor, and weight files together. This repository is an artifact release, so the source project remains the authoritative reference for task-specific loading code.

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

This is a REAP (Router-weighted Expert Activation Pruning) compressed version of a Qwen3.5/3.6-35B-A3B hybrid linear-attention MoE (Heretic variant). 30% of the routed experts were pruned (256 → 180 per layer) using the Cerebras REAP saliency criterion.

Pruning was performed with the layer-wise (block-wise) calibration observer on a c4 calibration set, reap saliency, seed 42. The model is text-only (the multimodal vision tower of the base Omni model was dropped during pruning).

Evaluation

Measured with lm-evaluation-harness against this model served on vLLM (pipeline-parallel, BF16):

(30% of experts pruned; these scores indicate the model retains strong general knowledge and commonsense after compression.)

Architecture note (hybrid linear attention)

This is a hybrid model: 30/40 layers use Gated-DeltaNet linear attention (Mamba-style recurrent conv+SSM state) and 10/40 use full attention. Two consequences:

• Hugging Face transformers: loads and generates correctly on a single device. Do not use device_map="auto" to split it across multiple GPUs — splitting the recurrent linear-attention state across devices produces garbage. Use one GPU (if it fits) or CPU.
• vLLM: serve with pipeline parallelism across GPUs (--pipeline-parallel-size N --tensor-parallel-size 1). vLLM provides its own GatedDeltaNet kernels.

Method

REAP selects experts to prune that contribute minimally to the layer output by considering both the router gate-values and the average activation norm of the experts, preserving the router's independent control over the remaining experts (avoiding the functional-subspace collapse of expert merging).

Reproduce with cerebras/reap.