Grug 67B A2B Datakit SFT 262K — Special-token learning rate
Research artifact. This model has not been properly tested or evaluated and is not validated for production use. The 262K context length is a configuration value; long-context performance remains untested.
This repository contains a BF16 Grug mixture-of-experts checkpoint after supervised fine-tuning (SFT) on the agentic and reasoning mixture specified below. Long-context pretraining replay supplied 20% of scheduled token positions.
Training data
The run scheduled 67,108,864,000 token positions: 1,000 updates × 256 packed sequences × 262,144 tokens. The fixed mixture assigned 53,687,091,200 positions (80%) to SFT and 13,421,772,800 positions (20%) to long-context pretraining replay. It selected 203 SFT leaf sources. Sources smaller than one sample in a 64,000-sample mixture block were placed in a pooled component so they remained sampleable.
The SFT allocation by source family was:
| Source family | Allocated SFT tokens | Share of the complete run |
|---|---|---|
nemotron_sft_v3 | 36,794,071,444 | 54.827439% |
open_swe_traces | 5,070,685,440 | 7.555910% |
superior-reasoning | 3,594,385,286 | 5.356051% |
penfever-traces | 1,528,897,822 | 2.278235% |
massive_function_calling | 1,521,205,631 | 2.266773% |
synthetic-1 | 1,521,205,631 | 2.266773% |
coderforge | 1,305,253,275 | 1.944979% |
openthoughts4-code-glm-5.2-n4 | 1,014,137,088 | 1.511182% |
nemotron-terminal | 879,083,437 | 1.309936% |
numinamath-1.5 | 179,420,301 | 0.267357% |
ultrachat-persona-conversations | 163,778,531 | 0.244049% |
agenttrove-glm53-compactions | 95,601,464 | 0.142457% |
wildchat-glm53-format-completions | 9,506,138 | 0.014165% |
glm-5.2-kernelgym-rollouts | 8,154,853 | 0.012152% |
synthetic-misconceptions-conversations | 1,704,859 | 0.002540% |
| Long-context pretraining replay | 13,421,772,800 | 20.000000% |
training_data_mix.json is the exact machine-readable specification. It records all 203 SFT leaf allocations, the excluded-source list, the replay recipe, and the 356 top-level weights saved in the finished W&B run. The checked in allocation and the W&B configuration agree for every unpooled SFT source; the pooled SFT weight also agrees exactly.
Model and provenance
| Property | Value |
|---|---|
| Architecture | GrugMoeForCausalLM (grug_moe) |
| Parameters | 67,078,882,816 total; approximately 2B active non-embedding parameters per token |
| Experts | 256 routed experts, top-4 routing, plus a shared expert |
| Layers / hidden size | 26 / 2,560 |
| Attention heads / KV heads | 20 / 5 |
| Vocabulary size | 128,256 |
| Configured context length | 262,144 tokens |
| Sliding window | 2,048 tokens |
| QK multiplier | 1.75 |
| Checkpoint step | 158,000 |
| SFT updates | 1,000, starting from base step 157,000 |
| Export | BF16, 39 safetensors shards |
The starting point was an unpublished base checkpoint at step 157,000. Training resumed its optimizer state without warmup. The peak base learning rate was 5e-5; updates to seven special-token rows in both input embeddings and the LM head were scaled by sqrt(32), giving an effective peak learning rate of 2.828427e-4 for those rows. Router weights remained trainable. Router biases and deferred load-balancing bias updates stayed frozen.
Seven LM-head rows were initialized from single-token anchors: 128006 from 3560 ( role), 128007 from 1984 ( message), 128002 from 1781 ( think), 128003 from 4320 ( answer), 128005 from 5507 ( tool), 128011 from 842 ( end), and 128009 from EOS (128001). Only those LM-head optimizer moments were reset. Input embeddings and their optimizer moments were preserved at initialization and remained trainable. BOS and EOS were preserved; no new tokens were added. The SFT data loader enforced a one-epoch limit.
Training run: grug-67b-sft-20260920-special-token-lr-frozen-bias-1000-cpfix.
Seven attention layers use full causal attention; the other 19 use the sliding window.
Chat and generation settings
Omitting enable_thinking defaults to thinking. For a vLLM chat request, select nonthinking mode with:
{"chat_template_kwargs": {"enable_thinking": false}}
For local prompt formatting:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"open-athena/Grug-67B-A2B-Datakit-SFT-262K-2026.09.21",
)
prompt_ids = tokenizer.apply_chat_template(
[{"role": "user", "content": "What is 17 times 23?"}],
tokenize=True, return_dict=False,
add_generation_prompt=True, enable_thinking=False,
)
The exported generation configuration specifies both <|end_of_text|> (128001) and <|eot_id|> (128009) as stop IDs. The tokenizer's EOS remains 128001. Preserve both generation stop IDs when configuring another runtime.
Thinking output uses <|start_think|> (128002) and <|end_think|> (128003). Decode with skip_special_tokens=False when these boundaries are needed. Tools use JSON inside <tool_call>...</tool_call>; input arguments may be objects or JSON strings. Tool-call delimiters remain ordinary text, not newly added special tokens.
The inference template adds the thinking default to the training template. Export checks cover default and explicit thinking modes, assistant text beside tool calls, and equivalent formatting for string messages and messages represented as lists of text parts.
Serving
This architecture requires a runtime with Grug MoE support. No serving runtime has been validated for this checkpoint. Stock Transformers model loading has not been validated.
Evaluation and limitations
This model is a research artifact and has not been properly tested or evaluated. No inference smoke test or benchmark result is established for this checkpoint. Training loss and export-format checks do not establish generation quality, safety, or numerical equivalence to the native checkpoint.
License
The model materials are released under the OpenMDW License Agreement, version 1.1.