jojangju-KR-31B
Korean multimodal reasoning on gemma-4-31B — text + image
🏠 moche.ai · 🎮 Playground · 📝 Blog · 💻 GitHub · 🤗 Org
🧭 Where this fits — the moche roadmap
moche-ai는 두 계보를 함께 굴립니다. 밑바닥 코어(moche-native)는 G0→G4로 성숙해가고, 병렬로 이 병합(merge) 라인이 강한 공개 base 위에서 한국어 멀티모달을 지금 당장 증명합니다 — 같은 옴니 목표로 수렴하는 제품 트랙.
✨ Highlights
- 🇰🇷 한국어 멀티모달 — 텍스트 + 이미지 추론
- 🧬 Merge line — gemma-4-31B 위 task arithmetic 조립
- 🏆 K-AI Leaderboard 제출 — 병합 라인의 실측 결과
- ⚡ 밑바닥 코어와 병렬 트랙 — 같은 옴니 목표
🚀 Quickstart
from transformers import AutoProcessor, AutoModelForImageTextToText
import torch
model_id = "moche-ai/jojangju-KR-31B"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto")
messages = [{"role": "user", "content": [
{"type": "image", "url": "https://example.com/photo.jpg"},
{"type": "text", "text": "이 이미지를 한국어로 설명해줘."},
]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Highlights
- Built on Gemma-4-31B — the latest Gemma generation (~31B dense, multimodal, 256K context).
- Large gains over the base on Korean cultural/commonsense (CLIcK) and reasoning (MuSR, Com2) axes, measured on a held-out split disjoint from any selection signal.
- Korean comprehension & reasoning focus; inherits Gemma-4 multimodality (text + image).
Specifications
| Item | Value |
|---|---|
| Parameters | ~31B (dense) |
| Architecture | Gemma4ForConditionalGeneration (gemma4) |
| Text layers / hidden / intermediate | 60 / 5376 / 21504 |
| Attention heads (Q / KV) | 32 / 16 |
| Modality | Text + Image (multimodal) |
| Context length | up to 262,144 tokens (256K) |
| Vocabulary | 262,144 |
| Precision | bfloat16 |
| Base | google/gemma-4-31B-it |
Training & Adaptation
적응·후처리 세부 사항은 비공개(proprietary)입니다. 가중치 수준의 후처리만 수행하며, 어떤 벤치마크 평가 데이터도 학습·병합에 사용하지 않습니다.
Adaptation and post-training details are proprietary and not disclosed in this release. Only weight-level post-processing is performed, and no benchmark evaluation data is used in the process.
Evaluation
Protocol: generative multiple-choice, log-probability argmax over answer tokens,
apply_chat_template, single-GPU (bf16 / 4-bit nf4). Numbers below are from an internal
proxy evaluation on a held-out split (kept strictly separate from any selection signal).
Official K-AI Leaderboard (operated by AI Hub / NIA on non-public benchmark data) numbers are
pending submission and are authoritative.
아래 수치는 자체 로컬 프록시(홀드아웃) 결과이며, 공식 K-AI Leaderboard(AI Hub/NIA 운영, 비공개 벤치마크) 점수와 다를 수 있습니다. 공식 수치가 정본입니다.
| Benchmark | gemma-4-31B-it (base) | jojangju-KR-31B |
|---|---|---|
| CLIcK (한국어 문화·상식, full) | 0.840 | 0.980 |
| MuSR (다단계 추론) | 0.490 | 0.635 |
| Com2 (인과·상식 추론) | — | 0.716 |
| KMMLU* (지식, 자체 홀드아웃) | 0.646 | 0.663 |
* KMMLU-Pro–style reproduction in our own harness — not the official private leaderboard set. Official K-AI Leaderboard numbers are pending submission.
No benchmark test data is used in adaptation or merging. The held-out split is kept strictly separate from any selection signal to avoid selecting on the eval.
Usage
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
import torch
model_id = "moche-ai/jojangju-KR-31B"
processor = AutoProcessor.from_pretrained(model_id)
model = Gemma4ForConditionalGeneration.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": [{"type": "text", "text": "대한민국 행정법의 기본 원칙을 설명해줘."}]}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt",
).to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
객관식 평가 시에는 고정 디코딩(greedy)을 권장합니다. 모델은 Gemma-4 chat template을 사용합니다.
Memory note. The 262,144-token vocabulary makes the output logits large. For single-GPU eval/serving prefer 4-bit (nf4) loading and
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
Intended Use
- 한국어 지식/상식/추론 질의응답, 객관식(MCQ) 평가, 연구·벤치마크 용도
- Korean knowledge/commonsense/reasoning QA, MCQ evaluation, research and benchmarking
Limitations
- 사실 오류·환각·편향이 있을 수 있으며, 안전 정렬은 기반 모델 수준에 의존합니다.
- 벤치마크 점수가 실제 사용 품질을 보장하지 않습니다.
- May produce factual errors, hallucinations, or bias; safety alignment inherits from the base model. Benchmark scores do not guarantee real-world quality.
License
Built on a Gemma-family architecture and distributed under the Gemma Terms of Use. By using this model you agree to the Gemma license terms.
Acknowledgements
Base model: google/gemma-4-31B-it.