Teutonic-II 110B Genesis
Teutonic-II 110B Genesis is an open-weight, compute-efficient sparse Mixture-of-Experts (MoE) language model with approximately 110B total parameters and an estimated ~7.3B activated parameters per token. The Genesis checkpoint is initialized with random weights and has not yet undergone pretraining, providing a clean starting point for continued pretraining and decentralized model improvement through Teutonic on Bittensor Subnet 3.
The model is built on a modified MiMoV2-style architecture, combining sparse MoE layers with hybrid Global and Sliding Window Attention, Grouped Query Attention, asymmetric QK/V head dimensions, partial Rotary Position Embeddings, and attention-sink mechanisms. In addition to adopting MiMoV2 architectural concepts, the Teutonic team introduces a modification to the MoE implementation that enables expert sharing.
1. Introduction
Teutonic-II 110B Genesis is designed as a compute-efficient large-scale sparse language model architecture for continued pretraining and decentralized model improvement.
The model uses a custom MiMoV2-style architecture with several efficiency-oriented components:
- Sparse Mixture-of-Experts (MoE): 256 routed experts are available in each MoE layer, while only 8 routed experts are selected per token.
- Shared Expert: Each MoE layer additionally includes one shared expert.
- Expert Sharing: Teutonic introduces an architectural modification that enables expert sharing within the MoE implementation.
- Hybrid Attention: The network combines Global Attention with local Sliding Window Attention.
- Sliding Window Attention: SWA operates with a local window of 128 tokens.
- Grouped Query Attention: 48 query heads share 8 key/value heads.
- Asymmetric QK/V Dimensions: Query/Key heads use 192 dimensions while Value heads use 128 dimensions.
- Partial Rotary Position Embeddings: RoPE is applied to a fraction of the attention dimensions.
- Attention Sink Bias: Enabled for Sliding Window Attention layers.
- Sparse Activation: Only a small subset of the model's expert parameters is used for each token.
- BF16 Precision: The checkpoint configuration uses
bfloat16. - Long-Context Architecture: The configuration declares up to 1,048,576 positional embeddings.
The declared maximum position length is an architectural configuration value. It should not by itself be interpreted as evidence that this checkpoint has been trained or validated at the full 1M-token sequence length.
2. Model Summary
| Component | Teutonic-II 110B Genesis |
|---|---|
| Architecture | MiMoV2ForCausalLM |
| Total Parameters | ~110B |
| Estimated Activated Parameters / Token | ~7.3B |
| Precision | BF16 |
| Hidden Size | 3,072 |
| Number of Layers | 45 |
| Dense Layers | 1 |
| MoE Layers | 44 |
| Global / Full Attention Layers | 9 |
| Sliding Window Attention Layers | 36 |
| Attention Heads | 48 |
| Key / Value Heads | 4 |
| Q/K Head Dimension | 192 |
| Value Head Dimension | 128 |
| Dense FFN Intermediate Size | 16,384 |
| MoE Expert Intermediate Size | 1,024 |
| Routed Experts | 256 |
| Shared Experts | 1 |
| Expert Sharing | Enabled via Teutonic architectural modification |
| Routed Experts per Token | 8 |
| Router Scoring Function | Sigmoid |
| Expert Selection | Top-K |
| Top-K Method | noaux_tc |
| Sliding Window Size | 128 |
| Attention Chunk Size | 128 |
| Standard RoPE Theta | 10,000,000 |
| SWA RoPE Theta | 10,000 |
| Partial Rotary Factor | 0.334 |
| Vocabulary Size | 152,576 |
| Configured Max Positions | 1,048,576 |
| Input / Output Embeddings Tied | No |
| Attention Dropout | 0.0 |
| Activation Function | SiLU |