Grug 67B A2B Datakit SFT 262K — 2026.09.16
Research artifact. This model has not been properly tested or evaluated and is not validated for production use. The 262K context length is a configuration value; long-context performance remains untested.
This repository contains a BF16 Grug mixture-of-experts checkpoint after supervised fine-tuning (SFT) on a focused agentic and reasoning mixture, with a configured 20% share of packed sequences drawn from long-context pretraining replay. SFT sources include OpenThoughts code, SWE-ZERO, Penfever traces, NuminaMath, and GLM format-following and compaction data.
Model and provenance
| Property | Value |
|---|---|
| Architecture | GrugMoeForCausalLM (grug_moe) |
| Parameters | 67,078,882,816 total; approximately 2B active non-embedding parameters per token |
| Experts | 256 routed experts, top-4 routing, plus a shared expert |
| Layers / hidden size | 26 / 2,560 |
| Attention heads / KV heads | 20 / 5 |
| Vocabulary size | 128,256 |
| Configured context length | 262,144 tokens |
| Sliding window | 2,048 tokens |
| QK multiplier | 1.75 |
| Checkpoint step | 158,000 |
| SFT updates | 1,000, starting from base step 157,000 |
| Export | BF16, 39 safetensors shards |
The starting point was an unpublished base checkpoint at step 157,000. Training resumed its optimizer state without warmup. Five special-token embedding and output rows (128006, 128007, 128002, 128003, and 128009) were initialized from semantic text anchors, with their optimizer moments reset. BOS and EOS were preserved. No new tokens were added. The SFT mixture was capped at one pass per source.
Seven attention layers use full causal attention; the other 19 use the sliding window.
This is a separate SFT run from the September 11 release.
Chat and generation settings
Omitting enable_thinking defaults to thinking. For a vLLM chat request, select nonthinking mode with:
{"chat_template_kwargs": {"enable_thinking": false}}
For local prompt formatting:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"open-athena/Grug-67B-A2B-Datakit-SFT-262K-2026.09.16",
)
prompt_ids = tokenizer.apply_chat_template(
[{"role": "user", "content": "What is 17 times 23?"}],
tokenize=True, return_dict=False,
add_generation_prompt=True, enable_thinking=False,
)
Generation stops on both <|end_of_text|> (128001) and <|eot_id|> (128009). The tokenizer's EOS remains 128001. Preserve both generation stop IDs when configuring another runtime.
Thinking output uses <|start_think|> (128002) and <|end_think|> (128003). Decode with skip_special_tokens=False when these boundaries are needed. Tools use JSON inside <tool_call>...</tool_call>; input arguments may be objects or JSON strings. Tool-call delimiters remain ordinary text, not newly added special tokens.
The inference template adds the thinking default to the training template. Export checks cover default and explicit thinking modes, assistant text beside tool calls, and string/text-part formatting parity.
Serving
This architecture requires a runtime with Grug MoE support. The previous release was served with Marin's vLLM fork at f0d7cc7f5874. This checkpoint has not yet been tested in that runtime. Stock Transformers model loading has not been validated.
Evaluation and limitations
This model is a research artifact and has not been properly tested or evaluated. No inference smoke test or benchmark result is established for this checkpoint. Training loss and export-format checks do not establish generation quality, safety, or numerical equivalence to the native checkpoint.
License
The model materials are released under the OpenMDW License Agreement, version 1.1, using the same license text as the September 11 release.