SJ-ko

A 75M parameter Korean GPT built and trained from scratch. Current release: SJ-0.7.1simft (sj-0.7.1simft.gguf). The earlier 11M releases sj-0.5.6simf.gguf and sj-0.5.1si.gguf are kept in legacy/.

In training: a ~400M-parameter successor (run28 — 26 layers · 1024 dim, 32k byte-fallback BPE, 7.6B-token Korean/English/code corpus) on Kaggle T4×2. It is not published yet; the training history below is updated as runs finish.

No Hugging Face Transformers, no existing training framework. The BPE tokenizer, attention, training loop, and GGUF converter were all written by hand. This is an educational project, not a production model. It invents facts and can fall into repetition loops.

• Live demo: http://49.50.140.57:3000 (still serves the previous 11M release, sj-0.5.6simf)
• Author: Sungjin Moon, Seoul National University, Department of Physics & Astronomy, Physics
• Contact: mongsil88@snu.ac.kr

Specification

The GGUF was checked against the PyTorch model: greedy generation matches token for token on 9 of 9 test prompts (Ollama 0.33.3).

Usage (Ollama)

ollama create sj-0.7.1simft -f Modelfile
ollama run sj-0.7.1simft

For the previous 11M release, download legacy/sj-0.5.6simf.gguf and legacy/Modelfile-0.5.6simf into one folder and run ollama create sj-0.5.6simf -f Modelfile-0.5.6simf.

The bundled Modelfile carries settings that are required. Without them the model echoes the question back or never stops generating.

TEMPLATE """{{- range .Messages }}{{- if eq .Role "user" }}<|user|>{{ .Content }}<|assistant|>{{- else }}{{ .Content }}<|endoftext|>{{- end }}{{- end }}"""
PARAMETER stop "<|endoftext|>"
PARAMETER stop "<|user|>"

The template walks .Messages, so multi-turn conversations work through /api/chat. Building the prompt yourself with raw: true does not work — the special tokens are then treated as plain text rather than control tokens.

Tools

This release was fine-tuned to call three tools — calc, clock, and search (BM25 over a local document index) — through the special tokens <|tool|>, <|end|>, and <|result|>. In this "forced" variant the calling program, not the model, decides when a tool is needed and writes the tool token; the model then fills in the arguments (for calc) and answers from the tool's result.

<|user|> question <|assistant|> <|tool|> name args <|end|> <|result|> result <|end|> answer <|endoftext|>

Plain ollama run never inserts a tool token, so there the model answers from its own weights only. Tool calls need a harness that runs this loop; the harness is not part of this repository.

Self-hosting the chat UI

serve_ollama.py is a dependency-free chat server (Python standard library only, no PyTorch). It talks to a local Ollama instance and streams responses.

OLLAMA_HOST=127.0.0.1:11434 ollama serve &
ollama create sj-0.7.1simft -f Modelfile
python3 serve_ollama.py --model sj-0.7.1simft --host 0.0.0.0 --port 3000

Keep Ollama bound to localhost. Its API has no authentication and allows model deletion, so it must never be exposed to the internet.

Training data

Pre-training (corpus v5, 2.63B unique characters; share of the training stream, with repeats)

Supervised fine-tuning (55,588 pairs loaded; 72 longer than 1,024 tokens dropped, 499 held out for validation)

Attribution

This model was developed using the following datasets provided by AI Hub (https://aihub.or.kr):

• 018.논문자료 요약 데이터
• 029.대규모 구매도서 기반 한국어 말뭉치 데이터
• 121.한국어 성능이 개선된 초거대AI 언어모델 개발 및 데이터

It was also trained on 나무위키 (namu.wiki) text, licensed CC BY-NC-SA 2.0 KR, and on Wikipedia text, licensed CC BY-SA.

No original data is included in this repository, and none is redistributed.

License

CC BY-NC-SA 4.0 for sj-0.7.1simft.gguf, because its training data includes 나무위키 text under a NonCommercial-ShareAlike license. Non-commercial use only; derivatives must carry the same license.

The earlier 11M releases (legacy/sj-0.5.6simf.gguf, legacy/sj-0.5.1si.gguf) did not use 나무위키 and were published under MIT; they remain available under MIT.

The terms of the original training data remain with their respective providers.

Training history

Every run from the first character-level bigram to the current release. Loss per char (nats per character) is the only column comparable across runs — token loss depends on the tokenizer, which changed several times. Notes are the author's own working notes, in Korean.