PARSER NOTE: reasoning sits between <|channel>thought and <channel|>; the visible answer follows <channel|> in the same turn The chat template is not stock Gemma 4 Upstream Gemma 4 appends an empty thought channel (<|channel>thought\n<channel|>) to non-thinking turns. That form shows up 0 times in 1,000 training turns of this corpus — a no-thoughts turn simply carries no channel — so the template here drops it. The stock template ships alongside as chat_template.gemma-it-original.jinja. Restore it and you push the model out of distribution: reasoning leaks into the answer and tool calls lose their opener. What the thoughts look like depends on the system prompt. Under a SillyTavern-style character card the model writes a structured planner (~750 chars; 23/23 of the cards that opened a channel). Under the corpus's own RP framing it writes short first-person interiority (~90 chars). The model learned both forms separately, and the prompt picks which one you get. The planner, when it shows up: SCENE: where/when, atmosphere, key environmental details currently in play CHARACTERS: who is present and their current physical/emotional state and motivation CONTINUITY: established facts that must stay consistent THREADS: active tensions and where they stand right now PLAN: what THIS turn needs to accomplish and the approach it takes One more thing to expect: a conversational companion persona usually produces no thought channel at all (0/6 in testing), even with thinking on. Companion rows in the corpus are mostly non-thinking, and the model follows the data. Tool calling Gemma 4 tool calls use a DSL, not JSON: FORM: <|tool_call>call:NAME{key:<|"|>value<|"|>}<tool_call|> EXAMPLE: <|tool_call>call:get_weather{city:<|"|>Lisbon<|"|>}<tool_call|> Serve tool calls inside one turn In the training corpus a whole tool episode lives inside a single <|turn>model, with <|tool_response> blocks interleaved inline. The model never emitted <turn|> after a call, so it never learned to yield there. A harness that waits for <turn|> will hang while the model keeps generating plausible calls — the classic infinite tool loop. SERVE WITH: stop=["<tool_call|>"] THEN: inject <|tool_response>response:NAME{...}<tool_response|> and continue the SAME turn NEVER: wait for <turn|> after a tool call Key Details BASE MODEL: google/gemma-4-12B LICENSE: gemma NOTE: the base is multimodal, so the checkpoint loads with AutoModelForImageTextToText (see Quickstart) Training METHOD: FULL-PARAMETER SFT -- ms-swift (swift sft), DeepSpeed ZeRO-2, torch SDPA attention, custom liger fused CE STAGES: three, each tagged in this repo; main = stage 3
stage 1 (v1-baseline-rp) aviary burn corpus, 1 epoch 1,917 steps @ lr 1e-5 eval 1.311 tok-acc 0.6465 stage 2 (v2-corrected-rp) thinking-weighted resample 568 steps @ lr 2e-6 eval 1.3067 tok-acc 0.6479 stage 3 (main) + 4,000 converted RP-reasoning rows 574 steps @ lr 2e-6 eval 1.301 tok-acc 0.6491 Why there is a stage 3 Stage 2 could think, but only under one prompt shape: 19,605 of the corpus's 20,666 thought-bearing rows share a single RP framing. So the model opened a thought channel on 8/8 in-corpus rows — and on 1 of 25 real character cards. Stage 3 mixed in RP-reasoning rows under ~4,000 distinct character cards so that thinking no longer depends on one specific prompt. opens a thought channel on 25 held-out character cards (846-5,053 chars, short opening message): stage 2: 1/25 (4%) stage 3: 23/25 (92%)
unchanged across the pass: stop rate 10/10 in both thinking and non-thinking modes tool-call round trip passes stray channels with thinking off: 0/3 P(<channel|>) at the true close: 1.000 Sampling Use the defaults in generation_config.json. "temperature": 1.0, "top_k": 64, "top_p": 0.95, Quickstart from transformers import AutoModelForImageTextToText, AutoTokenizer tok = AutoTokenizer.from_pretrained("aimeri/spoomplesmaxx-whiskeyjack-12B") model = AutoModelForImageTextToText.from_pretrained( "aimeri/spoomplesmaxx-whiskeyjack-12B", dtype="bfloat16", device_map="auto") msgs = [{"role": "user", "content": "Solve (x + 2)^2 = 0."}] ids = tok.apply_chat_template(msgs, add_generation_prompt=True, enable_thinking=True, return_tensors="pt").to(model.device) out = model.generate(ids, max_new_tokens=512) print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=False)) This one will hear how unhinged you are