Grug 67B A2B Datakit SFT 262K — Head-only initialization, scaled LR
Research artifact. This model has not been properly tested or evaluated and is not validated for production use. The 262K context length is a configuration value; long-context performance remains untested.
This repository contains a BF16 Grug mixture-of-experts checkpoint after supervised fine-tuning (SFT) on a focused agentic and reasoning mixture, with a configured 20% share of packed sequences drawn from long-context pretraining replay. SFT sources include OpenThoughts code, SWE-ZERO, Penfever traces, NuminaMath, and GLM format-following and compaction data.
Model and provenance
| Property | Value |
|---|---|
| Architecture | GrugMoeForCausalLM (grug_moe) |
| Parameters | 67,078,882,816 total; approximately 2B active non-embedding parameters per token |
| Experts | 256 routed experts, top-4 routing, plus a shared expert |
| Layers / hidden size | 26 / 2,560 |
| Attention heads / KV heads | 20 / 5 |
| Vocabulary size | 128,256 |
| Configured context length | 262,144 tokens |
| Sliding window | 2,048 tokens |
| QK multiplier | 1.75 |
| Checkpoint step | 158,000 |
| SFT updates | 1,000, starting from base step 157,000 |
| Export | BF16, 39 safetensors shards |
The starting point was an unpublished base checkpoint at step 157,000. Training resumed its optimizer state without warmup, with both optimizer settings (learning_rate and adam_lr) scaled from 5e-5 to 2.828427e-4 (sqrt(32)). This scaling applies globally. “Head-only” describes reinitialization; non-router model weights remained trainable. Five LM-head token rows were copied from single-token anchors: 128006 from 3560 ( role), 128007 from 1984 ( message), 128002 from 1781 ( think), 128003 from 4320 ( answer), and 128009 from EOS (128001). Only those LM-head optimizer moments were reset. Input embeddings and their optimizer moments were preserved at initialization and remained trainable. Router matrices, router biases, and pending load-balancing biases stayed frozen. BOS and EOS were preserved; no new tokens were added. The SFT mixture was capped at one pass per source.
Training run: grug-67b-sft-20260917-head-only-sqrt32-frozen-router-1000.
Seven attention layers use full causal attention; the other 19 use the sliding window.
This is a separate SFT run from the September 11 release.
Chat and generation settings
Omitting enable_thinking defaults to thinking. For a vLLM chat request, select nonthinking mode with:
{"chat_template_kwargs": {"enable_thinking": false}}
For local prompt formatting:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"open-athena/Grug-67B-A2B-Datakit-SFT-262K-2026.09.17",
)
prompt_ids = tokenizer.apply_chat_template(
[{"role": "user", "content": "What is 17 times 23?"}],
tokenize=True, return_dict=False,
add_generation_prompt=True, enable_thinking=False,
)
The exported generation configuration specifies both <|end_of_text|> (128001) and <|eot_id|> (128009) as stop IDs. The tokenizer's EOS remains 128001. Preserve both generation stop IDs when configuring another runtime.
Thinking output uses <|start_think|> (128002) and <|end_think|> (128003). Decode with skip_special_tokens=False when these boundaries are needed. Tools use JSON inside <tool_call>...</tool_call>; input arguments may be objects or JSON strings. Tool-call delimiters remain ordinary text, not newly added special tokens.
The inference template adds the thinking default to the training template. Export checks cover default and explicit thinking modes, assistant text beside tool calls, and string/text-part formatting parity.
Serving
This architecture requires a runtime with Grug MoE support. The previous release was served with Marin's vLLM fork at f0d7cc7f5874. This checkpoint has not yet been tested in that runtime. Stock Transformers model loading has not been validated.
Evaluation and limitations
This model is a research artifact and has not been properly tested or evaluated. No inference smoke test or benchmark result is established for this checkpoint. Training loss and export-format checks do not establish generation quality, safety, or numerical equivalence to the native checkpoint.
License
The model materials are released under the OpenMDW License Agreement, version 1.1, using the same license text as the September 11 release.