Qwen3.8-Flash-Next-RAZOR-96B-A6B-E384of512
Qwen3.8-Flash-Next with 25% of its routed experts removed using RAZOR. Each main MoE layer retains 384 of 512 experts. The separate MTP expert bank is also reduced to 384 experts, using the unobserved-layer policy described below.
No gradient updates, recovery training or new quantization were applied. The surviving expert weights and router rows are slices of the base checkpoint; non-target tensor bytes are unchanged.
| Property | Base checkpoint | This checkpoint |
|---|---|---|
| Routed experts per MoE layer | 512 | 384 |
| Routed experts removed | 0% | 25% |
| Main language model, excluding n-gram tables, MTP and vision | 125,743,653,760 (~125.744B) | 95,528,936,320 (~95.529B) |
| N-gram embedding tables, stored separately | 51,200,245,760 (~51.200B) | 51,200,245,760 (~51.200B), unchanged |
| Distinct stored MTP parameters | 2,607,150,848 (~2.607B) | 1,977,677,568 (~1.978B) |
| Vision encoder | 448,931,056 (~0.449B) | 448,931,056 (~0.449B), unchanged |
| Complete stored floating-point parameters | 179,999,981,424 (~180.000B) | 149,155,790,704 (~149.156B) |
| Activated parameters, using the base model's reported convention | ~6B | ~6B |
| Routed experts per token (top-k) | 10 | 10, unchanged |
| Shared experts per main MoE layer | 1 | 1, unchanged |
| Main MoE decoder layers / stored MTP MoE layers | 48 / 1 | 48 / 1 |
| Weight dtype | BF16 | BF16 |
Naming scope: 96B rounds the main language-model count, including token embeddings and the LM head but excluding n-gram lookup tables, MTP and the vision encoder. The n-gram weights are still included in the download: they have not been deleted. HF's automatic parameter badge counts the full stored checkpoint and may therefore show approximately 149B, rather than the 96B used in the name. The 35 integer buffer elements are not learned parameters.
The official card separates its headline 125B language model with 6B activated from 51B n-gram and 4B MTP. The table above follows that separation but counts distinct stored tensors exactly. In particular, the official revision contains 2.607B distinct MTP tensor elements; the approximate 4B headline is not substituted for that storage count. A6B retains the official activated-parameter convention and is not a new throughput or FLOP measurement.
The other released budget is Qwen3.8-Flash-Next-RAZOR-65B-A6B-E256of512.
Selection and calibration
- Main layers: rank experts by the RMS of the local output change when a selected expert is removed and the router promotes its highest-ranked unselected expert. Keep the highest-scoring 384 experts per layer.
- Aggregation: RMS conditional on the calibration tokens routed to each expert, not a corpus sum and not an arithmetic mean.
- Calibration evidence: 2,151,120 valid tokens packed into 66 rows of up to 32,768 tokens, covering all 48 main MoE layers. Existing additive statistics were reused to recompute the Refill-RMS selection; this was not a fresh collection on the currently released RazorCal.
- Corpus: the historical
MoECal_v2.5experiment corpus. Do not interpret this release as having been calibrated on the current public RazorCal.calibration_info.jsonrecords the statistics-pack digest and calibration geometry. No calibration examples are included here. - MTP: the ordinary text calibration does not execute
mtp.layers.0. Its experts were selected by the L2 norm of their original router rows (mtp_policy=router_norm), an explicit heuristic rather than measured RAZOR saliency. Its keep-set is namedmtp_0.
This release is a newly exported Refill-RMS checkpoint, not the older CDMp2/LOO checkpoint with a changed label. The 25% main-layer selection differs from that older experiment by 309 retained expert choices across the 48 layers.
The base revision is de4b8e4d43b917e7706784d8bb445c9af86a3540. kept_expert_indices.json gives original expert indices in output order. pruning_info.json records method=razor, aggregation=rms, the full keep-set and the MTP policy. These artifacts allow the exact slicing operation to be inspected independently of the calibration data.
Native architecture and loading
The official architecture is qwen4_exp / Qwen4ExpForConditionalGeneration, not qwen3_next. Keep the original configuration: substituting an older Qwen architecture is not supported. The conditional model preserves the vision branch, hyper-connections, PLE/n-gram embeddings, linear attention and QSA.
The validation environment used Transformers 5.16.1 and BF16/FP32 native kernels. Transformers 5.8.1 does not contain this model implementation. Use AutoModelForImageTextToText for the conditional checkpoint rather than allowing an automatic causal-LM loader to drop the vision branch. The original tokenizer, chat template, generation defaults and image/video preprocessing configurations are included.
Approximately 298.31 GB are required for tensor payloads alone in BF16, before activations, caches, conversion buffers and runtime overhead. In particular, the large n-gram embedding affects device placement. This release does not prescribe a verified full-size serving topology. Optional FLA/causal-convolution kernels may require CUDA even when the model code has a PyTorch fallback; use a compatible native environment.
The RAZOR codebase describes the pruning method. Verification of this release used a Qwen3.8-capable development checkout; a published package or earlier Git revision without qwen4_exp support cannot verify it merely by accepting its file paths.
Verification and limitations
Completed checks for this checkpoint:
- All 1,658 stored tensors were checked against the source and keep-set: all surviving expert rows and router rows in matching order, all non-target tensor bytes, the configuration, and the complete keep manifest.
- License bytes and tokenizer/template assets were checked; rewritten JSON auxiliary files were compared by their complete parsed contents, not formatting.
- A real-pretrained-weight comparison of the first native decoder layer on two short text batches matched an independently sliced source reference with zero difference.
- Separate tiny-native-model regressions covered FP32/BF16 routing and refill, PLE/ngram, hyper-connections, QSA, streaming windows, query chunking and export/reload logits. Those tests are not full-size benchmarks.
verification_report.json records storage and auxiliary checks. release_info.json records the public naming, parameter scope and additional verification boundaries.
Not established: full-size end-to-end log-probability equivalence, benchmark retention, multimodal task quality, long-context generation quality or MTP speculative-decoding behavior. The prefix check is not a whole-model forward check. Expert pruning is lossy; successful slicing verification does not guarantee accuracy, robustness, termination behavior or performance without regression. Evaluate on the intended workload before deployment.
License and attribution
The weights are a derivative of Qwen3.8-Flash-Next and remain subject to Qwen Community License 1.0, included verbatim as LICENSE. This is not Apache-2.0. The license includes conditions for large commercial services and separate-license requirements for certain commercial Model-as-a-Service and AI Work Assistant uses; consult the complete upstream terms rather than assuming unrestricted commercial hosting.
RAZOR contributes expert selection and slicing, not newly trained weights. Cite the RAZOR paper for the pruning method and the official Qwen3.8-Flash-Next model card for the base model.