Kisoku 1.6B Preview

Kisoku is a 1.6 billion parameter language model trained from scratch by 0ARCH. It is not a fine-tune of another company's model: the weights were initialized randomly and pretrained on Google TPUs, then chat-tuned.

This is a preview. The base model has finished stage 1 of a planned 3-stage pretraining run (about 417B of the planned ~510B tokens). Stages 2 and 3 add more math and code and the learning-rate cool-down; the full release will follow when they finish. The preview already holds a conversation, follows instructions, and stops cleanly at the end of its turn.

Ollama and llama.cpp builds (GGUF) are at 0arch-io/kisoku-1.6b-preview-GGUF.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("0arch-io/kisoku-1.6b-preview")
model = AutoModelForCausalLM.from_pretrained("0arch-io/kisoku-1.6b-preview", torch_dtype="auto")

messages = [{"role": "user", "content": "Explain why the sky is blue in two sentences."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**inputs, max_new_tokens=256)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For Ollama and llama.cpp, see 0arch-io/kisoku-1.6b-preview-GGUF, which has ready-made GGUF files and a Modelfile.

Model

Training

Pretraining (stage 1 of 3). 199,000 steps of 2.1M tokens, about 417B tokens in total, on a TPU v4-32 with MaxText. Data mix: Nemotron-CC 35%, Ultra-FineWeb 35%, StarCoder 16%, FineMath 6%, OpenWebMath 4%, MegaMath-Web-Pro 4%. Muon optimizer for matrices and AdamW for embeddings and norms, peak learning rate 3e-4, weight decay 0.1.

Chat tuning. 432,702 conversations (303M tokens), two passes, learning rate 5e-5 with cosine decay, loss on assistant replies only:

Before training, every conversation was filtered: any assistant turn containing a refusal or an "as an AI language model" disclaimer was dropped, as were answers where the assistant claimed to be another model.

Uncensored

Kisoku is trained to answer directly rather than refuse or moralize. It has no safety tuning. You are responsible for how you use it and for what you build with it.

Example outputs

Greedy decoding, no system prompt:

Who are you? I'm Kisoku, an open-source language model made by 0ARCH. I was trained from scratch, not fine-tuned from another company's model.

Explain why the sky is blue in two sentences. The sky is blue because the Earth's atmosphere scatters shorter wavelengths of light, such as blue and violet, more than longer wavelengths like red and orange. This scattering effect, known as Rayleigh scattering, is responsible for the blue appearance of the sky during the day.

Write a Python function that checks whether a number is prime.

def is_prime(n):
    if n < 2:
        return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0:
            return False
    return True

Limitations

• Math and multi-step reasoning are weak. Asked "3 apples, buy 5 more, eat 2", it answered 11 instead of 6. Stages 2 and 3 of pretraining target this.
• Specialized knowledge is thin and it will confidently make things up, as any 1.6B model does. Check facts that matter.
• English only. 4096-token context.
• No safety tuning (see above).

License

The Kisoku weights are released under the MIT license. The tokenizer is Meta's Llama 3.2 tokenizer, which is covered by the Llama 3.2 Community License. The chat-tuning data includes no_robots, which is licensed CC BY-NC 4.0, and SmolTalk2 and Hermes-3, which are Apache 2.0; check the dataset licenses if you plan commercial use.

Acknowledgements

Trained with Cloud TPUs from Google's TPU Research Cloud.