GPT-OSS-20B-Dense-4B-ACP-K4
Research artifact, not a general-purpose model. These weights are released so that the experiments in the paper Pruning and Distilling Mixture-of-Experts into Dense Language Models can be reproduced. Each student is distilled on a fixed, small token budget (~4B tokens for Qwen3, 0.3B tokens for DeepSeek-V2-Lite and GPT-OSS) purely so that expert scoring and grouping methods can be compared under an equal budget. Absolute quality is therefore far below the teacher and below pretrained models of the same size, and no instruction tuning or alignment was applied. Please do not use this as an off-the-shelf assistant.
What this is
A dense student obtained by pruning and distilling a Mixture-of-Experts teacher, from the paper Pruning and Distilling Mixture-of-Experts into Dense Language Models. Code: https://github.com/krafton-ai/moe-to-dense
Results
Full comparison group for this architecture, GPT-OSS-20B to dense, 0.3B-token distillation (paper Table 18). This model's row is in bold, and rows whose weights are also released link to them.
| Configuration | Wino | Hella | ARC-E | ARC-C | MMLU | Avg |
|---|---|---|---|---|---|---|
| SF, K=4 | 51.6 | 29.4 | 34.0 | 22.4 | 23.3 | 32.15 |
| SF, K=8 | 51.6 | 29.3 | 33.3 | 21.7 | 22.8 | 31.72 |
| CP, K=4 | 50.9 | 30.4 | 36.4 | 23.0 | 23.6 | 32.86 |
| CP, K=8 | 49.7 | 29.3 | 32.2 | 23.2 | 23.1 | 31.49 |
| ACP, K=4 | 53.0 | 31.9 | 35.6 | 23.0 | 23.3 | 33.36 |
| ACP, K=8 | 53.1 | 30.5 | 33.6 | 23.3 | 23.7 | 32.82 |
| DO-ACP, K=4 | 53.0 | 32.1 | 36.7 | 23.2 | 23.7 | 33.71 |
| DO-ACP, K=8 | 51.3 | 29.9 | 33.5 | 22.5 | 23.3 | 32.11 |
| Random FFN + teacher attn | 50.2 | 27.5 | 28.5 | 23.0 | 23.2 | 30.46 |
| Random initialization | 50.0 | 26.0 | 25.8 | 25.3 | 23.0 | 30.02 |
| Teacher (GPT-OSS-20B) | 59.3 | 39.9 | 80.9 | 53.7 | 49.6 | 56.67 |
Downstream accuracy is Winogrande 5-shot, HellaSwag 10-shot, ARC-Easy 25-shot, ARC-Challenge 25-shot and MMLU 5-shot. Avg is the unweighted mean of the five benchmarks. The teacher is a post-trained reasoning model evaluated in completion mode for consistency with the students, which understates its native-format capability (MMLU 49% in completion mode versus 72% with the chat template).
Configuration
| Field | Value |
|---|---|
| Teacher | openai/gpt-oss-20b |
| Student parameters | 4.19B |
| Distillation data | FineWeb-Edu (sample-10BT), 0.3B tokens |
| Expert scoring | ACP |
| Experts kept (K) | 4 |
| Grouping | pure pruning (K equals the teacher's top-k, one expert per group) |
| Down-projection scaling | proportional |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("jhyuckkim/GPT-OSS-20B-Dense-4B-ACP-K4", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("jhyuckkim/GPT-OSS-20B-Dense-4B-ACP-K4")
The student is a GptOssForCausalLM with num_local_experts=1, so GPT-OSS's
gated activation, per-layer biases, sliding-window attention and YaRN RoPE are
preserved. Load it with that architecture class rather than a generic dense
one.
Citation
@article{kim2026pruning,
title={Pruning and Distilling Mixture-of-Experts into Dense Language Models},
author={Kim, Junhyuck and Yun, Jihun and Kim, Haechan and Kim, Gyeongman and Bae, Joonghyun and Cho, Jaewoong},
journal={arXiv preprint arXiv:2605.28207},
year={2026}
}