⚠️ EXPERIMENTAL WEIGHTS — NOT A FINAL RELEASE ⚠️

[!WARNING]

THESE ARE EXPERIMENTAL RESEARCH WEIGHTS. THIS IS NOT A FINISHED OR PRODUCTION MODEL.

Every qwen-3.8-next-40b-exp-* repository is an intermediate checkpoint of an ongoing weight-surgery research line — not a release, not a final version, not a validated product.

• Not validated for end-user deployment, standalone inference, or any downstream product.
• No safety, alignment, or quality guarantees. Behaviour can change or regress between checkpoints.
• Non-standard architecture (qwen4_exp): requires an inference stack that explicitly supports it.
• Published only for reproducibility of MoE pruning, router-healing, PLE-calibration and distillation experiments.

If you are looking for a model to actually use, use the upstream Qwen release instead.

Qwen-3.8-Next-40B-MoE-Healed (BF16)

Full BFloat16 weights of the experimental hybrid model Qwen-3.8-Next-40B after memory-pruning surgery and a routing-stabilization run (router healing) over the 48 MoE gates $W_{\text{gate}}$, with the PLE associative memory table pruned to 2.0M entries.

1. Diagnosis of the pruned base

The pruned base checkpoint showed expert starvation and routing collapse:

1. Routing imbalance — the 48 $W_{\text{gate}}$ routers still carried logits initialized for a much larger expert space, leaving ~38% of experts fully inactive or below 0.2% traffic.
2. Syntactic collapse into loops — critical layers (layer 3, Gini 0.728; layer 44, Gini 0.706) concentrated up to 8.8% of all traffic in a single monopolist expert, inducing infinite lexical repetition in open-ended generation.
3. PLE hypertrophy — factual-memory injection ran at $\rho = |\Delta h_{\text{PLE}}| / |h_{\text{residual}}| = 0.658$, roughly double the stable range ($0.20 \sim 0.35$).

2. Router-healing SFT (150 steps)

A closed-loop protocol with triple balancing pressure:

• ST-MoE auxiliary balance loss: $\mathcal{L}{\text{aux}} = N \sum{i=1}^N f_i \cdot P_i$
• Logarithmic starvation barrier (KL): asymmetric exponential penalty on experts whose share falls below $\frac{0.5}{N}$.
• Inter-expert load variance: $\mathcal{L}_{\text{var}} = \text{Var}(f)$.
• Parametric surgery: 99.96% of the model frozen. Only the 48 $W_{\text{gate}}$ routers (15.73M trainable parameters) were optimized with AdamW, warmup and cosine decay ($LR_{\max} = 3 \times 10^{-4}$).

3. Quantitative results, pre vs post healing

Audited on an NVIDIA RTX PRO 6000 (96 GB VRAM) with a multi-domain battery (concurrent Python, CoT mathematics, legal doctrine, tool calling):

[!NOTE] These numbers describe relative improvement over a broken checkpoint, not competitive benchmark results. No standard benchmark (MMLU, GSM8K, HumanEval…) has been run on this checkpoint, and none should be inferred from the table above.

Architecture (qwen4_exp)

• Total parameters: 40.22B (1,294 tensors, 80.44 GB BF16)
• Activated per token: top-10 routed experts + 1 shared dense expert ($\approx$ 2.6B expert parameters per token, on top of the attention and embedding stack)
• Checkpoint class: text-only (Qwen4ExpForCausalLM); the vision tower is not included
• PLE memory: n-gram table pruned to 2,000,000 entries

4. Requirements

Non-standard architecture. It needs an inference stack with explicit qwen4_exp support (Gated DeltaNet linear attention + 4-stream Hyper-Connections + PLE associative memory). Standard dense-Transformer loaders will not work.

[!NOTE] Shard 1 is ~49.6 GB. Use hf download / snapshot_download with resume rather than a plain browser download.

Experiment line

5. Credits

Based on the architectures and research of David Aylward (logic65), the Qwen team (Alibaba Cloud), and work on adaptive MoE routing.

License

Apache 2.0, inherited from the upstream Qwen base model.