Gemma-4-12B-EsperGrug

Gemma-4-12B-EsperGrug is an exquisite, mathematically calibrated fusion of two specialized Gemma-4-12B fine-tunes:

1. ValiantLabs/gemma-4-12B-it-Esper4: An agentic software engineering, systems architecture, DevOps, and MLOps specialist (distilled via DeepSeek-V4-Pro).
2. kai-os/Grug-12B: A compact-reasoning model optimized for high-density logical deduction, strict invariant tracking, and minimal token overhead.

By blending Grug's dense, verification-first cognitive core with Esper4's agentic execution engine, this model solves difficult multi-step problems with rigorous logic and directly produces robust, production-grade code without conversational filler.

Ecosystem & Companion Repositories

• Quantized GGUF Weights:
  For quantized deployment on local edge hardware or llama.cpp (including Q5_K_L, Q4_K_M, and Q8_0), use the official GGUF repository:
  👉 OliviaRossi/gemma-4-12B-EsperGrug-GGUF

• Recommended Chat Template & Harness:
  To fully unlock the model's dual-system reasoning and agentic tool-calling capabilities, use the recommended Jinja chat template and orchestration configuration:
  👉 OliviaRossi/Gemma-4-Chat-Template-Tri-Craft-Harness

Architectural & Merging Methodology

Rather than performing simple linear weight averaging (which causes attention norm degradation and neuron cancellation), Gemma-4-12B-EsperGrug was synthesized from scratch via a custom, low-RAM streaming pipeline using:

1. Vectorized Row-Wise Hyperspherical Linear Interpolation (SLERP)

All multi-dimensional attention projections ($Q, K, V, O$) and embedding matrices were interpolated row-by-row along their unit hypersphere geodesics ($S^{D-1}$), followed by Euclidean norm conservation: $$\Omega = \arccos\left(\frac{u \cdot v}{|u|_2 |v|_2}\right), \quad \text{SLERP}(u, v; t) = \frac{\sin((1-t)\Omega)}{\sin\Omega}u + \frac{\sin(t\Omega)}{\sin\Omega}v$$ This eliminates destructive collinear interference and preserves activation variances across all 48 layers.

2. DARE-Tuned Sparsification for MLP Knowledge Blocks

Feed-forward associative layers (gate_proj, up_proj, down_proj) were merged using DARE (Drop and REscale) magnitude-thresholding. Pruning the bottom 20% insignificant parameter deltas removes structural cross-talk between Grug's symbolic logic pathways and Esper4's CLI/DevOps memory.

3. Cognitive-Executive Curvature Depth Schedule ($t(l)$)

Weight blending follows a non-linear curvature across the 48 transformer layers:

• Layers 0–10 (Syntactic Foundation): Balanced $50/50$ blend ensuring stable lexical representation and base grammar.
• Layers 11–32 (Deep Reasoning Core): Dips to 65% Grug-12B ($t = 0.35$). Infuses the internal representations with dense chain-of-thought, invariant checking, and algorithmic constraint propagation.
• Layers 33–47 (Executive Action Tail): Ramps up to 70% Esper4 ($t = 0.70$). Anchors top-layer decoding to agentic tool use, bash/infrastructure syntax, shell pipelines, and executable code generation.

Quickstart & Usage

Standard Inference (transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliviaRossi/gemma-4-12B-EsperGrug"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "Analyze the race conditions in this concurrent worker pool and refactor it in Rust:"
messages = [{"role": "user", "content": prompt}]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        inputs,
        max_new_tokens=1024,
        temperature=0.2,
        top_p=0.95,
        repetition_penalty=1.05
    )

print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Loading with the Recommended Tri-Craft Chat Template

For advanced agentic loops, tool integration, and specialized reasoning delimiters, pull the chat template from the companion harness:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("OliviaRossi/gemma-4-12B-EsperGrug")

# Load and apply the recommended Tri-Craft template
harness_tokenizer = AutoTokenizer.from_pretrained("OliviaRossi/Gemma-4-Chat-Template-Tri-Craft-Harness")
tokenizer.chat_template = harness_tokenizer.chat_template

Behavioral Characteristics

• Concise Thinking Traces: Avoids redundant conversational preambles ("Sure, I can help with that!"); begins reasoning and code synthesis immediately.
• Invariant & Edge-Case First: Inherits Grug's rigorous constraint auditing, checking bounds, types, and nil-states before writing logic.
• Production-Grade Infrastructure: Outputs robust shell, Docker, Terraform, Kubernetes, Python, and Rust scripts directly aligned with modern CI/CD standards.

Credits & Attribution

This model is built upon the foundational work of the open-source community:

• Base Model: Google Gemma-4
• Esper4 Distillation & Agentic Tuning: ValiantLabs
• Grug Compact Reasoning Tuning: kai-os
• Merge Architecture & Engineering: Olivia Rossi