Moderato-V1-Pro (113.3B Sparse MoE)

Following intensive research into Sparse Mixture-of-Experts stability and trajectory divergence, we are pleased to introduce Moderato-V1-Pro, a flagship open-weights MoE model engineered for state-of-the-art coding, systems engineering, and agentic workflows.

Built upon 6 specialized 27B expert domains fused at the feed-forward network (FFN) layer with shared attention backbones, Moderato-V1-Pro introduces Reflexive Role Routing (RRR) — a 2-level hierarchical meta-controller that continuously evaluates hidden-state divergence and executes hot-swap expert interventions during generation without KV-cache loss.

🌟 Moderato-V1-Pro Highlights

• Reflexive Role Routing (RRR): 2-level hierarchical routing combining static token-level softmax gating G(x) with checkpointed divergence probes pθ(ht, g) at N=64 token intervals to intercept hallucinations before failure cascades occur.
• 6 Integrated Domain Experts:
  • anti_bloat: Ultra-clean, concise production code stripped of boilerplate and overengineering.
  • clean_diffs: Surgical git unified diff patches with line-level boundary preservation.
  • deep_math_cot: Formal Olympiad-level mathematical reasoning and multi-step Chain-of-Thought.
  • systems_rust: Low-level systems engineering, lock-free concurrency, and memory-safe Rust idioms.
  • modern_apis: Modern cloud/SWE architectures, asynchronous web frameworks, and schema-strict REST/gRPC APIs.
  • agentic_fable: Autonomous multi-step planning, tool orchestration, and recursive self-reflection.
• Sparse MoE Efficiency: 113.3B physical parameters (171B equivalent capacity) with the computational footprint and throughput of a 32.7B active model.
• Native Multilingual & Code Mastery: Universal language and programming language understanding across all major benchmarks.
• Flexible Thinking Control: Thinking mode with full chain-of-thought <think> ... </think> blocks, tuneable with reasoning_effort.

📐 Model Overview

• Model Type: Sparse Mixture-of-Experts (MoE) Causal Language Model
• Routing Mechanism: Reflexive Role Routing (RRR) Hierarchical Meta-Controller
• Total Parameters: 113.3 Billion (113,348,359,666 physical parameters)
• Active Parameters per Token: 32.7 Billion (Top-2 Experts)
• Number of Transformer Layers: 64
• Hidden Dimension (dmodel): 5120
• Feed-Forward Intermediate Dimension: 17,408
• Number of Dedicated Experts per Layer: 6 FFN Expert Pathways
• Attention Configuration:
  • Query Heads (Q): 40
  • Key/Value Heads (KV): 8 (Grouped-Query Attention - GQA)
  • Head Dimension: 128
  • Rotary Position Embedding (RoPE): Base frequency 1,000,000
• Context Length: 131,072 tokens natively, extensible up to 1,000,000 tokens via YaRN RoPE scaling.

📊 Benchmark Results

Performance Evaluation Matrix

🔬 Scientific Innovation: Reflexive Role Routing (RRR)

Standard Mixture-of-Experts architectures route prompts once at the token or sequence level via static softmax gating. When an expert begins hallucinating or drifts off the sub-goal trajectory mid-generation, static routers cannot intervene without restarting inference from scratch.

Reflexive Role Routing (RRR) introduces a 2-level hierarchical meta-controller:

1. Level 1 (Static MoE Gate)

Evaluates input embedding x to compute soft top-K expert weights (K=2): $$G(x) = \text{Softmax}\left(\text{TopK}(W_g x + \epsilon, k=2)\right)$$

2. Level 2 (Checkpointed Divergence Probe)

Every N = 64 tokens, a lightweight probe pθ(ht, g) analyzes the current hidden state ht against the trajectory sub-goal g, predicting divergence δ ∈ [0, 1] and confidence c ∈ [0, 1]:

• δ < 0.3: CONTINUE — proceed on the fast path.
• δ ≥ 0.3, c ≥ 0.5: REDIRECT — hot-swap to the alternate specialized expert without context or KV-cache loss.
• c < 0.5: ESCALATE — early escape to meta-orchestrator.

🚀 Quickstart & Inference

Serving with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nitrai-research/Moderato-V1-Pro"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

prompt = "<|im_start|>user\nImplement a lock-free bounded queue in Rust with atomic memory ordering.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

High-Throughput Serving (vLLM & SGLang)

# Launch with vLLM
vllm serve nitrai-research/Moderato-V1-Pro \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --max-model-len 131072

Local Inference with llama.cpp & Ollama (GGUF)

Moderato-V1-Pro can be converted and deployed locally using llama.cpp and Ollama via GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0):

[!NOTE] How Reflexive Role Routing operates in llama.cpp / Ollama:

• Standard GGUF Execution (Level 1 MoE): In pure C++ execution engines (llama.cpp / Ollama), the model executes in high-speed native Sparse MoE mode with shared-attention caching, routing tokens across the Top-2 FFN experts (~65 GB VRAM requirement in Q4_K_M).
• Full Dynamic Hot-Swapping (Level 2 RRR): The full probe divergence controller (pθ(ht, g) real-time trajectory steering) is supported in Python runtimes (transformers, vLLM, sglang) and custom inference sidecars.

Running with llama.cpp:

# Run with llama-cli in 4-bit quantization
llama-cli -m Moderato-V1-Pro-Q4_K_M.gguf \
    -p "<|im_start|>user\nWrite a lock-free bounded queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
    -ngl 99 --ctx-size 32768

Running with Ollama:

Create a Modelfile:

FROM ./Moderato-V1-Pro-Q4_K_M.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 32768
PARAMETER stop "<|im_end|>"
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

Then compile and run:

ollama create moderato-v1-pro -f Modelfile
ollama run moderato-v1-pro

📜 Citation & License

@misc{nitrai2026moderatov1pro,
  title={Moderato-V1-Pro: Reflexive Role Routing in 113.3B Sparse Mixture-of-Experts},
  author={NitrAI Research Team},
  month={August},
  year={2026},
  publisher={Hugging Face}
}

Licensed under the Apache 2.0 License.