CalmaCatCoder-Next-Mini

A tiny experimental chat model (96.6M parameters) trained completely from scratch on one consumer GPU (RTX 5060 Ti, 16 GB) by ViorikaAI. It is a hobby project: a small calm, cat-themed Python helper that knows who it is, speaks ChatML and writes very simple Python.

Please read the limitations below. This is a learning project, not a useful assistant. It memorised a handful of templates and will confidently produce wrong code and wrong facts.

Model details

The tokenizer is byte-level

Token id equals the byte value, so "def " becomes [100, 101, 102, 32] and one word costs several tokens. This is why the context is short. <|im_start|> and <|im_end|> are plain text made of bytes, not special tokens, so the model ends its answer by writing the literal text <|im_end|>. Stop generation on that string (see the example below).

Chat format (the same text the model was trained on, available as tokenizer.apply_chat_template):

<|im_start|>system
You are CalmaCatCoder-Next, a calm and friendly cat-themed coding assistant. You explain things step by step and say so when you are not sure.<|im_end|>
<|im_start|>user
Who are you?<|im_end|>
<|im_start|>assistant

If you pass no system message, the one above is inserted automatically. Other system prompts were never seen in training, so expect them to be mostly ignored.

Quick start

Requires transformers>=4.51 (Qwen3 support). No trust_remote_code needed.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, StoppingCriteria, StoppingCriteriaList

repo = "ViorikaAI-org/CalmaCatCoder-Next-mini"      # or a local folder
device = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).to(device, dtype=torch.bfloat16).eval()

END = tok.encode("<|im_end|>")                      # the end marker is ordinary bytes
class StopOnEnd(StoppingCriteria):
    def __call__(self, input_ids, scores, **kwargs):
        return torch.tensor([input_ids[0, -len(END):].tolist() == END] * input_ids.shape[0],
                            device=input_ids.device)

messages = [{"role": "user", "content": "Who are you?"}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=300,
                         stopping_criteria=StoppingCriteriaList([StopOnEnd()]))
print(tok.decode(out[0, inputs["input_ids"].shape[1]:]).replace("<|im_end|>", "").strip())

Sampling defaults live in generation_config.json (temperature 0.7, top-k 40). Keep prompt + answer under 1024 bytes; the model never saw longer sequences.

How it was trained

All of it ran on a single RTX 5060 Ti (16 GB) under Windows with a PyTorch nightly build.

Stage 1: pre-training from scratch (1 epoch, ~7.6 hours). 29,908 steps × 32,768 byte-tokens ≈ 980M tokens, about 36k tokens/s. AdamW (lr 6e-4, betas 0.9/0.95, weight decay 0.1), 300 warm-up steps, cosine decay to 10%, gradient clipping 1.0, bf16 autocast, torch.compile. Final validation loss 0.508 nats/byte on a held-out 2% of the mix. This number is not comparable with other models: it is per byte, and the held-out part contains near-copies of the templated persona dialogues.

Data mix (~1 GB, target shares; the exact proportions depend on what each source could supply):

• Python code: codeparrot/codeparrot-clean, the largest share (about 60%)
• English text: cosmopedia-v2 from HuggingFaceTB/smollm-corpus (15%)
• Math: meta-math/MetaMathQA (5%)
• ChatML dialogues: HuggingFaceTB/smol-smoltalk, only dialogues shorter than 1000 bytes (up to 20%)
• About 5 MB of synthetic persona dialogues generated by a script (identity, creator, greetings, a few Python tasks)

Stage 2: supervised fine-tuning (1 epoch, ~13 minutes). About 28k short ChatML dialogues (~17M tokens), loss computed on assistant tokens only, peak lr 1e-4. Mixture of programmatically generated dialogues (identity and creator questions in many phrasings, greetings, 18 small Python tasks, follow-ups such as "add a docstring", remembering a name within a chat), HuggingFaceTB/smoltalk (everyday-conversations) and a subset of smol-smoltalk. SFT validation loss was 0.123, which mostly reflects how templated that data is.

Intended use

Learning and curiosity: look at how a tiny model trained from scratch behaves, reuse the training recipe, test tokenizers or inference tooling on a very small model.

Not intended for real coding help, factual questions, any decision that matters, or production use.

Limitations (observed, not hypothetical)

• English only, almost only Python. Other languages and programming languages are outside what it learned.
• It memorised templates. For a coding request it picks the closest of the ~18 tasks it knows and adapts it. Real examples from testing:
  • "give me a random python code" produced a function named random_state that simply reverses a string, followed by an explanation copied from a different task.
  • "Write a function that multiplies all numbers in a list" produced a list comprehension that filters even numbers.
  • After "My name is Zorro." and one more turn, "What is my name?" was answered with a different name (Sam).
• Facts are unreliable, and it often confidently continues with plausible-looking nonsense.
• Short memory. 1024 bytes of context covers a couple of turns at most.
• No safety training of any kind. It is too small to be dangerous, but it has no guardrails either.
• Very low training loss does not mean good behaviour here: it mostly measures memorised templates.

Training data notes and license

Part of the training data (cosmopedia-v2, smol-smoltalk, smoltalk and the synthetic persona dialogues) was generated by other language models, so this model inherits their style and any of their errors. The code corpus consists of public repositories with mixed licenses. Check each dataset card for its license before reusing the weights or the data mix.

License: CalmaCat Open License (CCPL-1.0)