🏛️ QU-SSM-130M-MoE: Continuous Quasi-Unitary State Space Model with Sparse Mixture-of-Experts

📄 Official Research Paper (PDF) & Open Verification

Title: "Gated Quasi-Unitary Lie-Algebra Recurrent State Space Models"
Author: Prannessh K.V.A. (@prannesshkva)

📥 Download Full Research Paper (PDF) | 🏛️ Zenodo DOI: 10.5281/zenodo.22283431 | 🎮 Live Interactive Space

QU-SSM-130M-MoE is a continuous quasi-unitary state space sequence architecture co-designed with sparse SwiGLU Mixture-of-Experts (MoE). Designed and invented by Prannessh K.V.A. (Sole Architect & Inventor), QU-SSM eliminates monotonic exponential memory dissipation present in classical real-valued SSMs while delivering strictly constant O(1) step memory (0.19 MB) and 3.32x generation speedup over modern baseline Transformers.

🧬 Base Model Lineage & Technical Notes

• Core Architecture: Novel Continuous Quasi-Unitary Lie-Algebra Recurrent State Space Operator (SO(2) phase rotations) with 8 SwiGLU Mixture-of-Experts and Top-2 gating.
• Tokenizer Lineage: Standard GPT-2 Byte-Pair Encoding (BPE) vocabulary (50,257 tokens).
• Pre-training & Calibration: Initialized and pre-trained on roneneldan/TinyStories (~20M+ tokens) to empirically validate non-dissipative recurrent memory and narrative coherence.
• Parameter Footprint: 134.89M Total Parameters, 78.27M Active Parameters per token.
• State Space Dynamics: Strictly preserves vector norms (||R(theta)||_2 = 1.000) through dual-trigonometric rotation, eliminating contractive gradient decay across long sequence horizons.
• Official Research Contact: LinkedIn — Prannesh K. V. A.

🔍 What is QU-SSM (QU-SSM-130M-MoE)?

QU-SSM is a state-of-the-art linear-time sequence engine that formulates state transitions as continuous Lie-group rotations over SO(2) ≅ U(1). Unlike classical real-decay state space models that suffer from exponential forgetting, QU-SSM preserves state norm (‖R(θ)‖₂ ≡ 1.00000) while dynamically decoupling memory damping (γ_t).

Key Advantages of QU-SSM-130M-MoE:

• ⚡ 3.32x Faster Inference: Generates at 5.55 tok/s (180.16 ms/tok step latency) compared to 1.67 tok/s for standard dense Transformer baselines.
• 💾 Constant O(1) Memory: Requires strictly 0.19 MB of state RAM regardless of context length (1,894x smaller than a 360 MB KV-cache at L = 8,192).
• 🧠 42% Lower Active Compute: Evaluates only 78.27M active parameters per token across 8 SwiGLU experts with Top-2 routing.
• 🌊 Lowest Spectral Phase Distortion: Demonstrates 1.9648 rad phase error on continuous sensor and financial telemetry streams.

🏆 SOTA Benchmark Leaderboard (~135M Scale)

🔬 Mathematical Formulation of QU-SSM

The core recurrence equation evolves according to:

h_t = γ_t · R(θ_t) · h_{t-1} + u_t

Where:

• R(θ_t) ∈ SO(2) is a 2x2 rotation matrix with strict norm preservation: ‖R(θ_t)‖₂ ≡ 1.00000.
• γ_t = σ(W_γ · x_t + b_γ) ∈ (0, 1) is the decoupled memory gate controlling damping independently.

Exact Real Dual-Component Parallel Prefix Scan:

S = cumsum(log γ_t).clamp(min=-12.0, max=0.0)
Φ = cumsum(θ_t)
h_t = exp(S) · [ cos(Φ) · cumsum(u_real) - sin(Φ) · cumsum(u_imag) ]

💻 How to Install and Run QU-SSM

Run QU-SSM-130M-MoE directly using standard Hugging Face transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Prannesshkva/QU-SSM-130M-MoE"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "Once upon a time in a world of state space models,"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50)
print(tokenizer.decode(output[0]))

🌐 Universal Multimodal Support in QU-SSM

The QU-SSM backbone supports four continuous and discrete sequence interfaces:

❓ Frequently Asked Questions (FAQ)

What makes QU-SSM different from classical real-decay SSMs?

Classical SSMs enforce negative real decay (A ∈ ℝ⁻), creating a dissipative low-pass filter that loses oscillatory and high-frequency phase information (e^(-α·t) → 0). QU-SSM uses dynamic Lie-group rotations over SO(2) with ‖R(θ)‖₂ ≡ 1.00000, preserving state energy indefinitely while dynamically controlling damping via γ_t.

Is QU-SSM-130M-MoE an MoE model?

Yes. It co-designs the quasi-unitary recurrent state space engine with an 8-expert SwiGLU Mixture-of-Experts layer using Top-2 routing, evaluating only 78.27M active parameters per step.

What is the pretraining dataset for QU-SSM-130M-MoE?

This checkpoint is an early-stage foundational prototype pre-trained on ~20 Million tokens of structured narrative reasoning (TinyStories) over 5 epochs, achieving a 70x perplexity reduction (3,358 → 48.21 PPL).

🔒 Intellectual Property & Citation

• Sole Architect & Inventor: Prannessh K.V.A.
• Permanent Research DOI: 10.5281/zenodo.22283431
• License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
• Official Contact: LinkedIn — Prannesh K. V. A.

@software{prannesshkva_qu_ssm_2026,
  author       = {Prannesh K.V.A.},
  title        = {QU-SSM-MoE: Continuous Quasi-Unitary Lie-Group State Space Models with Sparse Mixture-of-Experts},
  month        = sep,
  year         = 2026,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.22283431},
  url          = {https://doi.org/10.5281/zenodo.22283431}
}

🔗 Related QU-SSM Models & Ecosystem Links

• 🚀 QU-SSM-130M-MoE (134.89M Flagship)
• ⚡ QU-SSM-60M-MoE (64.30M Mid-Tier)
• 🔬 QU-SSM-15M (29.80M Foundation)
• ⚛️ Liquid QU-SSM-130M (Continuous-Time LNN)
• 🎮 QU-SSM Scientific Benchmark Suite