⚠️ EXPERIMENTAL WEIGHTS — NOT A FINAL RELEASE ⚠️
[!WARNING]
THESE ARE EXPERIMENTAL RESEARCH WEIGHTS. THIS IS NOT A FINISHED OR PRODUCTION MODEL.
Every
qwen-3.8-next-40b-exp-*repository is an intermediate checkpoint of an ongoing weight-surgery research line — not a release, not a final version, not a validated product.
- Not validated for end-user deployment, standalone inference, or any downstream product.
- No safety, alignment, or quality guarantees. Behaviour can change or regress between checkpoints.
- Non-standard architecture (
qwen4_exp): requires an inference stack that explicitly supports it.- Published only for reproducibility of MoE pruning, router-healing, PLE-calibration and distillation experiments.
If you are looking for a model to actually use, use the upstream Qwen release instead.
Qwen-3.8-Next-40B-Exp — Clean Healed (BF16)
Post-surgery healed version of Qwen3.8-Flash-Next-Uncensored after pruning 75% of the experts
(128 surviving experts per layer, top-10 active).
1. Surgical protocol
- Analytic PLE pre-attenuation (layer 1): factor $\kappa = 0.6955$ applied to
norm_queryandnorm_keyto restore the in-vivo ratio $\rho \approx 0.250$. - Decoupled router-healing SFT:
- 48 MoE gates retrained with $\mathcal{L}_{\text{ST-MoE}} (\alpha=0.01) + \mathcal{L}_z (\beta=10^{-4})$.
- 60 steps of continuous convergence (loss 0.0145; layer-23 Gini 0.276).
- No memory leakage and no damage to the 99.96% of the network left frozen.
2. Post-healing evaluation
Qualitative probes in direct mode (enable_thinking=False):
| Probe | Result |
|---|---|
| Stop-token behaviour | Emits <|im_end|> within 4 tokens (Confirmado.<|im_end|>) |
| Structured Python | 93% Distinct-4, clean syntax, docstrings, $O(N)$ complexity notes |
| Spanish procedural law | 97.8% Distinct-4, native lexicon, doctrinally accurate |
| Negative constraints | 100% compliance |
[!NOTE] These are internal qualitative probes on a handful of prompts, not benchmark scores. No standard benchmark has been run on this checkpoint.
Architecture (qwen4_exp)
| Property | Value |
|---|---|
| Total layers | 48 hybrid MoE blocks |
| Hidden size | 2,560 |
| Attention | Gated DeltaNet linear attention, 3 linear blocks per 1 full-attention block (full_attention_interval: 4) |
| Hyper-Connections | 4 residual streams, rank-320 projections |
| Routed experts | 128 per layer, top-10 active (num_experts_per_tok: 10) |
| Shared expert | 1 dense expert (intermediate size 640) |
| PLE memory | Associative n-gram table, 2,000,000 base entries (ngram_size: 3) |
| Vocabulary | 248,320 |
| Native context | 262,144 tokens |
| Precision | BFloat16 |
- Checkpoint class: text-only (
Qwen4ExpForCausalLM); the vision tower is not included - Weights: 17 BF16 shards, 1,294 tensors, 80.44 GB
3. Requirements
Non-standard architecture. It needs an inference stack with explicit qwen4_exp support
(Gated DeltaNet linear attention + 4-stream Hyper-Connections + PLE associative memory).
Standard dense-Transformer loaders will not work.
Repository contents
| File | Notes |
|---|---|
model-*-of-00017.safetensors | 17 BF16 shards |
model.safetensors.index.json | Weight map |
DISECCION_MATEMATICA_POST_HEALING.json | Post-healing routing/statistics dissection |
config.json, generation_config.json, chat_template.jinja, tokenizer.json | Configuration and tokenizer |
Experiment line
| Repository | Role |
|---|---|
qwen-3.8-next-40b-exp-clean-bf16 | Clean baseline |
qwen-3.8-next-40b-exp-clean-healed-bf16 | Router-healed, text-only |
qwen-3.8-next-40b-exp-moe-pruned-bf16 | 512→128 expert pruning |
qwen-3.8-next-40b-exp-moe-healed-bf16 | Pruned + router-healed + PLE 2.0M |
qwen-3.8-next-40b-exp-ple-2m-table | Routing tables and calibration artifacts |
qwen-3.8-next-40b-exp-cartridge-general | Delta cartridge testbed |
qwen-3.8-next-40b-exp-profiling-metrics | Profiling metrics |
License
Apache 2.0, inherited from the upstream Qwen base model.