Abliteration parameters
| Parameter | Value |
|---|
| direction_index | 26.50 |
| attn.o_proj.max_weight | 1.48 |
| attn.o_proj.max_weight_position | 27.79 |
| attn.o_proj.min_weight | 1.10 |
| attn.o_proj.min_weight_distance | 15.18 |
| mlp.down_proj.max_weight | 1.47 |
| mlp.down_proj.max_weight_position | 39.36 |
| mlp.down_proj.min_weight | 0.24 |
| mlp.down_proj.min_weight_distance | 17.75 |
Performance
INTELLECT-3-REAP-50
50% expert-pruned version of PrimeIntellect/INTELLECT-3 using Cerebras REAP (Router-weighted Expert Activation Pruning).
Model Details
| Property | Value |
|---|
| Base Model | PrimeIntellect/INTELLECT-3 (248B MoE) |
| Architecture | GLM-4 MoE (glm4_moe) |
| Compression | 50% (64 experts pruned) |
| Remaining Experts | 64 per layer |
| Parameters | ~124B |
| Format | BF16 SafeTensors |
| Size | 107 GB |
REAP Configuration
dataset: 0xSero/glm47-reap-calibration-v2
samples: 1360
- evol-codealpaca-v1: 700 (code generation)
- xlam-function-calling-60k: 330 (function calling)
- SWE-smith-trajectories: 330 (agentic multi-turn)
distance_measure: angular
seed: 42
model_max_length: 2048
compression_ratio: 0.50
prune_method: reap
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"0xSero/INTELLECT-3-REAP-50",
torch_dtype="auto",
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("0xSero/INTELLECT-3-REAP-50", trust_remote_code=True)
messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Related Models
| Model | Compression | Format | Size |
|---|
| INTELLECT-3-REAP-50 | 50% | BF16 | 107GB |
| INTELLECT-3-REAP-50-W4A16 | 50% | W4A16 GPTQ | ~30GB (coming soon) |
Citation
@article{cerebras2025reap,
title={REAP: Router-weighted Expert Activation Pruning for MoE Models},
author={Cerebras Systems},
year={2025}
}
Acknowledgments
This model was created as part of efficiency research for large MoE models.