alice-noa-chan/haru_3-student-base
Haru v3 student-base: 13,688,705 parameters, context 1024, full causal attention, QK RMSNorm, SwiGLU and an incremental KV cache. FFNs are independent per layer; linear adapter rank: 0. Training tokens: 480,509,952. Full FP32 weights are preserved. Variant: non-it. This is the compact student model. This model is trained for Korean children's-story continuation. General-language corpora support fluency, but story validation drives checkpoint selection.
Source: alice-noa-chan/haru.
Public implementation commit: 11f7c7bddfad1fb92234bc175ffb70499f39616e.
Teacher/student training uses the same tokenizer. No new external LLM data,
teacher logits or pretrained model weights were used. Existing synthetic corpora
were reused. Training corpora are not distributed with this model.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = 'alice-noa-chan/haru_3-student-base'
tokenizer = AutoTokenizer.from_pretrained(repo, revision='gated8-13m', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, revision='gated8-13m', trust_remote_code=True).eval()
inputs = tokenizer("작은 마을에 아침이 찾아왔어요.", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=120, use_cache=True)
print(tokenizer.decode(out[0], skip_special_tokens=True))
IT (chat) checkpoints also support tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False). Base checkpoints continue
text and are non-IT. Input plus generation must fit 1024 tokens.
Model weights and included source code are licensed under MIT (see LICENSE). Training-source records and raw corpora are kept outside the public release.
Validation Story BPC: 0.9490010008100048. Public benchmark comparisons and limitations are documented in the source repository. No claim of being the strongest under 18M is made without matching evidence.
Training and final test evaluation are complete. See evaluation.json for measurements and fixed continuations.
Tokenizer compatibility fix: implementation. Chat inputs include the training BOS token; skip_special_tokens=True removes role boundaries under Transformers 4 and 5. Trained weights are unchanged.
Released models and student candidates
Two students were compared at exactly 13,688,705 parameters and 500,039,680 completed distillation tokens each. Both are publicly available.
| Model | Role | Repository |
|---|---|---|
| Gated8-13m | Selected student, non-IT | haru_3-student-base |
| Gated8-13m IT | IT version of the selected student | haru_3-student-chat |
| PairShare8-r48 | Unselected experimental student, non-IT | haru_3-student-pairshare-base |
| Teacher base | Frozen non-IT teacher | haru_3-teacher-base |
| Teacher IT | Frozen IT teacher | haru_3-teacher-chat |
PairShare shares each SwiGLU FFN across two adjacent layers and adds a rank-48 linear residual adapter per layer. Its attention layers and KV caches remain independent. Gated8 uses independent FFNs with width 512; PairShare uses width 960.
| Candidate | Validation story BPC (lower is better) | Training-host CPU generation, 4 threads |
|---|---|---|
| PairShare8-r48 | 0.945834 | 93.28 tokens/s |
| Gated8-13m | 0.949001 | 110.63 tokens/s |
The paired document-bootstrap 95% interval for PairShare minus Gated validation BPC was [-0.009002, 0.003038], including zero. The predeclared selection rule therefore chose the faster CPU model, Gated8-13m. Test scores did not drive selection. This is one training seed; the bootstrap does not measure variation between training seeds. PairShare is preserved for architecture experiments; no PairShare IT checkpoint was trained.
See the Haru collection, training instructions, and completed evaluation.