alice-noa-chan/haru_3-student-chat

Haru v3 student-chat: 13,688,705 parameters, context 1024, full causal attention, QK RMSNorm, SwiGLU and an incremental KV cache. FFNs are independent per layer; linear adapter rank: 0. Training tokens: 10,092,544. Full FP32 weights are preserved. Variant: it. This is the compact student model. This model is trained for Korean children's-story continuation. General-language corpora support fluency, but story validation drives checkpoint selection.

Source: alice-noa-chan/haru. Public implementation commit: 11f7c7bddfad1fb92234bc175ffb70499f39616e. Teacher/student training uses the same tokenizer. No new external LLM data, teacher logits or pretrained model weights were used. Existing synthetic corpora were reused. Training corpora are not distributed with this model.

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = 'alice-noa-chan/haru_3-student-chat'
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "토끼가 친구를 만나는 동화를 이어 써 주세요."}],
    add_generation_prompt=True, return_tensors="pt", return_dict=True,
)
out = model.generate(**inputs, max_new_tokens=120, use_cache=True)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

IT (chat) checkpoints also support tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=False). Base checkpoints continue text and are non-IT. Input plus generation must fit 1024 tokens.

Model weights and included source code are licensed under MIT (see LICENSE). Training-source records and raw corpora are kept outside the public release.

Validation Story BPC: 1.0263175861794789. Public benchmark comparisons and limitations are documented in the source repository. No claim of being the strongest under 18M is made without matching evidence.

Training and final test evaluation are complete. See evaluation.json for measurements and fixed continuations.

IT inference example

import torch
messages = [{"role": "user", "content": "토끼가 친구를 만나는 동화를 이어 써 주세요."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.8, top_p=0.9, use_cache=True)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For GPU inference, move model and inputs to cuda. Input plus generated length must fit 1,024 tokens.

Tokenizer compatibility fix: implementation. Chat inputs include the training BOS token; skip_special_tokens=True removes role boundaries under Transformers 4 and 5. Trained weights are unchanged.

This experimental IT model remains weak at generalization: 53/70 exact matches for familiar entities/templates, 7/70 with new entities, and 0/70 when both entities and templates change. Actual chat-formatted fixed stories still lose objects, promises and event continuity; see the saved examples in evaluation.json.

Released models and student candidates

Two students were compared at exactly 13,688,705 parameters and 500,039,680 completed distillation tokens each. Both are publicly available.

PairShare shares each SwiGLU FFN across two adjacent layers and adds a rank-48 linear residual adapter per layer. Its attention layers and KV caches remain independent. Gated8 uses independent FFNs with width 512; PairShare uses width 960.

The paired document-bootstrap 95% interval for PairShare minus Gated validation BPC was [-0.009002, 0.003038], including zero. The predeclared selection rule therefore chose the faster CPU model, Gated8-13m. Test scores did not drive selection. This is one training seed; the bootstrap does not measure variation between training seeds. PairShare is preserved for architecture experiments; no PairShare IT checkpoint was trained.

See the Haru collection, training instructions, and completed evaluation.