arXiv: REAM: Merging Improves Pruning of Experts in LLMs

Qwen3-30B-A3B-Instruct-2507-REAM

This model is a compressed version of Qwen/Qwen3-30B-A3B-Instruct-2507. It is obtained by reducing the number of experts in each MoE layer from 128 to 96. This reduction is achieved by the REAM method described in https://bknyaz.github.io/blog/2026/moe/. The compressed model has 23B params (44GB) instead of 31B (57GB) of the original model, reducing storage and GPU memory requirements by roughly 25%. At the same time, the model retains >=94% of the original model's performance on a variety of benchmarks (see Evaluation section below). Additional efficiency optimization (e.g., quantization) can be added similarly to the original model.

Model Details

The model is exactly the same as Qwen/Qwen3-30B-A3B-Instruct-2507 except that number of experts is reduced from 128 to 96.

Evaluation

Model is evaluated using https://github.com/EleutherAI/lm-evaluation-harness/ except for LiveCodeBench, which is evaluated using https://github.com/LiveCodeBench/LiveCodeBench.

The following versions were used for eval:

• python >= 3.10
• torch : 2.7.1+cu126
• lm_eval : 0.4.9.1
• vllm : 0.10.1.1
• transformers : 4.57.1
• datasets : 3.2.0
• numpy : 1.26.4

For tasks IFEval, AIME25, GSM8K and HumanEval the following command was used for eval on 4xNVIDIA H100: python -m lm_eval --model vllm --model_args pretrained=${model},tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1,max_model_len=131072 --tasks ${task} --batch_size 1 --apply_chat_template=True --confirm_run_unsafe_code

For HumanEval, we use --task=humaneval_instruct.

For GPQA-Diamond, we add flags: --num_fewshot 5 --fewshot_as_multiturn and set --task=gpqa_diamond_n_shot.

For LiveCodeBench, we evaluate using: python -m lcb_runner.runner.main --model Qwen/Qwen3-30B-A3B-Instruct-2507 --scenario codegeneration --evaluate --local_model_path ${model} --release_version release_v6

For multi-choice question answering tasks (Winogrande, ARC-C, ARC-E, BoolQ, HellaSwag, MMLU, OpenBookQA, RTE), we evaluate using lm_eval on a single GPU with a batch size equal 16.

Other parameters are set to default.

Metrics

We report the metric from the first row printed by lm_eval.

For example, for IFEval, we report inst_level_loose_acc=0.8921 given the lm_eval's output:

Results

License

Please refer to the license of the original model Qwen/Qwen3-30B-A3B-Instruct-2507.