⚠️ EXPERIMENTAL WEIGHTS — NOT A FINAL RELEASE ⚠️
[!WARNING]
THESE ARE EXPERIMENTAL RESEARCH WEIGHTS. THIS IS NOT A FINISHED OR PRODUCTION MODEL.
Every
qwen-3.8-next-40b-exp-*repository is an intermediate checkpoint of an ongoing weight-surgery research line — not a release, not a final version, not a validated product.
- Not validated for end-user deployment, standalone inference, or any downstream product.
- No safety, alignment, or quality guarantees. Behaviour can change or regress between checkpoints.
- Non-standard architecture (
qwen4_exp): requires an inference stack that explicitly supports it.- Published only for reproducibility of MoE pruning, router-healing, PLE-calibration and distillation experiments.
If you are looking for a model to actually use, use the upstream Qwen release instead.
Qwen-3.8-Next-40B-Exp — 512→128 MoE Pruning (BF16)
Intermediate weights produced by a 512-to-128 MoE expert pruning surgery applied to
Qwen3.8-Flash-Next-Uncensored. This is the raw pruned state — before router healing.
Expect routing imbalance, expert starvation and repetition loops in open-ended generation;
the repaired checkpoint is
qwen-3.8-next-40b-exp-moe-healed-bf16.
Pruning method
- Pruning ratio: 512 router experts compressed to 128 surviving experts per layer (75% reduction).
- Selection: multi-stratum Pareto knapsack calibration over 5.7M tokens (CoT, coding, tool-calling and dialogue strata).
- Denoising: LASER SVD denoising plus cosine affinity matching between merged experts.
- Frozen: everything outside the expert banks.
Architecture (qwen4_exp)
| Property | Value |
|---|---|
| Total layers | 48 hybrid MoE blocks |
| Hidden size | 2,560 |
| Attention | Gated DeltaNet linear attention, 3 linear blocks per 1 full-attention block (full_attention_interval: 4) |
| Hyper-Connections | 4 residual streams, rank-320 projections |
| Routed experts | 128 per layer, top-10 active (num_experts_per_tok: 10) |
| Shared expert | 1 dense expert (intermediate size 640) |
| PLE memory | Associative n-gram table, 2,000,000 base entries (ngram_size: 3) |
| Vocabulary | 248,320 |
| Native context | 262,144 tokens |
| Precision | BFloat16 |
Checkpoint class: multimodal (Qwen4ExpForConditionalGeneration, 27-layer vision tower).
Repository contents
| File | Notes |
|---|---|
model-*-of-00131.safetensors | 104 BF16 shards, ~86 GB |
model.safetensors.index.json | Weight map (1,658 tensors, 80.83 GB) |
mascara_48_capas_optima.json | Per-layer surviving-expert mask (48 layers) |
DONE_CIRUGIA_MOE_COMPLETA.json | Surgery completion manifest |
calib_sota_gated_5m.parquet | Calibration corpus used for expert selection |
config.json, generation_config.json, chat_template.jinja, tokenizer.json | Configuration and tokenizer |
[!NOTE] Shard naming: files are named
-of-00131, but the checkpoint is complete with 104 shards — exactly whatmodel.safetensors.index.jsonreferences. The00131suffix is a leftover from the original sharding run; nothing is missing.
Known limitations
- Routing is unbalanced: ~38% of experts see little or no traffic, and some layers concentrate up to 8.8% of all tokens in a single expert.
- Open-ended generation can fall into lexical repetition loops.
- PLE memory injection ratio is roughly double the stable range.
Experiment line
| Repository | Role |
|---|---|
qwen-3.8-next-40b-exp-clean-bf16 | Clean baseline |
qwen-3.8-next-40b-exp-clean-healed-bf16 | Router-healed, text-only |
qwen-3.8-next-40b-exp-moe-pruned-bf16 | 512→128 expert pruning |
qwen-3.8-next-40b-exp-moe-healed-bf16 | Pruned + router-healed + PLE 2.0M |
qwen-3.8-next-40b-exp-ple-2m-table | Routing tables and calibration artifacts |
qwen-3.8-next-40b-exp-cartridge-general | Delta cartridge testbed |
qwen-3.8-next-40b-exp-profiling-metrics | Profiling metrics |
License
Apache 2.0, inherited from the upstream Qwen base model.