MediKo-9B-A4B (Instruct)
Korean medical language model built with VocMoE (DVE + MoDE). This is the inst (instruction-tuned (chat)) checkpoint of MediKo-9B-A4B.
| Spec | Value |
|---|---|
| Base | Qwen3-4B |
| Total / Active params | 9.4B / 4.5B |
| Medical experts (N) / top-K | 16 / 4 |
| Layers | 36 |
| Vocab (DVE-expanded) | 156,669 |
| Variant | instruction-tuned (chat) |
This checkpoint (zero-shot avg): KorMedMCQA 57.5 · KMMLU-Med 53.0 · EnMed 66.9 · KoBEST 63.9 · HAE-RAE 62.1
How it works
VocMoE combines two components:
- DVE (Domain Vocabulary Expansion) adds 5,000 Korean medical tokens (from KM-BERT, scored by fragmentation × frequency × PMI), cutting medical-term over-fragmentation ~64%.
- MoDE (Mixture-of-Domain-Experts) uses the vocabulary partition as the routing key: medical tokens (
id ≥ original_vocab_size) go to dedicated experts, general tokens keep the preserved dense FFN. The general path stays architecturally isolated, preventing catastrophic forgetting at the architectural level.
Family & benchmarks
| Model | Base | Total | Active |
|---|---|---|---|
MediKo-1.1B-A0.6B | Qwen3-0.6B | 1.13B | 0.59B |
MediKo-4B-A1B | Qwen3-1.7B | 3.49B | 1.69B |
MediKo-9B-A4B | Qwen3-4B | 9.4B | 4.5B |
MediKo-30B-A8B (not released) | Qwen3-8B | ~30B | ~8.1B |
Benchmark (zero-shot avg, instruction-tuned variants):
| Model | Active | KorMedMCQA | KMMLU-Med | EnMed | KoBEST | HAE-RAE |
|---|---|---|---|---|---|---|
| Qwen3-8B (ref) | 8B | 42.2 | 52.4 | 70.7 | 64.0 | 57.7 |
| KULLM3 (ref) | 10.7B | 42.7 | 39.1 | 65.4 | 67.7 | 54.0 |
| MediKo-1.1B-A0.6B | 0.59B | 32.6 | 33.3 | 51.1 | 53.4 | 43.1 |
| MediKo-4B-A1B | 1.69B | 44.8 | 44.5 | 60.8 | 59.3 | 50.2 |
| MediKo-9B-A4B | 4.5B | 57.5 | 53.0 | 66.9 | 63.9 | 62.1 |
Usage
⚠️ Custom architecture (
MediKoForCausalLM) — passtrust_remote_code=True. Token-type routing is generated insideforward()frominput_ids.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "MediKo/MediKo-9B-A4B-inst"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True,
torch_dtype=torch.bfloat16, device_map="auto",
)
messages = [{"role": "user", "content": "성인 2형 당뇨병의 1차 약물 치료는 무엇인가요?"}]
# Thinking ON (default). OFF: add enable_thinking=False
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
- Trained with a "Thinking Recovery" objective: supports both thinking (
<think>...</think>) and non-thinking modes. Sampling: thinkingtemp=0.6, top_p=0.95; non-thinkingtemp=0.7, top_p=0.8.
Intended use & limitations
Research/education for Korean medical NLP.
⚠️ Medical disclaimer. Not a medical device; research/informational use only. Outputs must not be used for diagnosis, treatment, prescription, or clinical decision-making. Always consult a qualified professional. May hallucinate; do not input personally identifiable patient data.
Limitations. Validated only on Korean medical text. MoE → total memory larger than a comparable dense model. CPT token budget (~19.5B) modest vs large English medical LLMs.
Training data
- Medical (Korean) ~108 GB + General (Korean) ~39 GB → ~19.5B tokens (≈76% / 24%).
- Pipeline: DVE → Qwen3→MediKo dense → MoE upcycle → CPT → SFT.
License
Apache-2.0 (inherits Qwen3 base license). Comply with applicable medical / privacy regulations in your jurisdiction.
Citation
Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026.
@inproceedings{cho2026vocmoe,
title = {VocMoE: Vocabulary-Guided Mixture-of-Experts for Korean Medical Language Models},
author = {Cho, Sangyeon and Han, Jaeho and Jeon, Mingyu and Kim, Junyeong},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026},
}