ISOM-130M-MoE: Ultra-Edge Deliberative Reasoning via Bounded-State SSM

Community Call-to-Action: If ISOM-130M-MoE accelerates your edge deployment or speculative decoding pipeline, please leave a Like (heart icon top-right)!

ISOM (Isometric State Operator Manifold) is a novel recurrent architecture built on Selective State Space Models (SSMs / Mamba), engineered to deliver long-context intelligence within a strictly bounded working memory — no KV-cache growth, no attention overhead, no OOM.

ISOM-130M-MoE is the ultra-compact prototype of the ISOM family — a 134.89M parameter (58.2M active) Mixture-of-Experts model designed for:

• 💾 Strict < 2 MB Working Memory: Unlike Transformers whose cache explodes, ISOM's recurrent state is strictly bounded below 2 MB regardless of sequence length (O(1) memory).
• ⚡ Ultra-Fast Speculative Drafter: Functions as a high-speed draft model for larger foundation models like ISOM-Qwen-1.5B-Instruct and ISOM-DeepSeek-Coder-V2-Lite, providing up to 2x - 3x speculative decoding speedups.
• 🧠 ARC & Math Deliberation: Optimized for multi-step deliberative reasoning tasks — Abstract Reasoning Corpus (ARC), symbolic math, and Chain-of-Thought generation — at the edge.
• 📱 Embedded AI Ready: Targets microcontrollers, mobile SoCs, Raspberry Pi, and any hardware where < 2 MB working memory is a hard constraint.

🧬 Base Model Lineage & Technical Notes

• Core Architecture: Continuous Quasi-Unitary Lie-Algebra Recurrent State Space Model (SO(2) phase rotations) coupled with 8 SwiGLU Mixture-of-Experts (Top-2 gating per forward step).
• Tokenizer Lineage: Standard GPT-2 Byte-Pair Encoding (BPE) vocabulary (50,257 tokens).
• Parameter Footprint: 134.89M Total Physical Parameters, 58.27M Active Parameters per forward token.
• State Dynamics: Constant O(1) working state memory strictly bounded below 2.0 MB across any sequence length without attention matrix materialization or KV-cache allocation.
• Primary Roles: High-speed edge speculative drafting for larger foundation backbones (ISOM-Qwen, ISOM-Falcon, ISOM-DeepSeek) and on-device deliberative reasoning for microcontrollers and embedded devices.
• Official Research Contact: LinkedIn — Prannesh K. V. A.

Unified ISOM Model Lineup

Memory Wall: Why O(1) Matters

Standard Transformer KV-cache grows linearly with context:

KV-cache memory = 2 * num_layers * num_kv_heads * head_dim * seq_len * sizeof(dtype)

At 128K tokens, Qwen2.5-Coder vanilla:
  = 2 * 28 * 2 * 64 * 131072 * 2 bytes = 3,584 MB (3.58 GB)

ISOM working state (any context length):
  = bounded constant (< 2 MB for 130M, 112 MB for 1.5B at 128K)
  = O(1) with respect to sequence length

ISOM eliminates this O(N) explosion entirely, making 128K context accessible on consumer GPUs.

Architecture

ISOM-130M-MoE replaces standard Transformer attention with ISOM recurrent layers — each a parameterized state-space operator that processes tokens without materializing any attention matrix. The MoE feed-forward layers (8 SwiGLU experts, TopK=2 active) maximize capacity while keeping activated compute minimal.

Model Specification

Deliberative Reasoning Benchmark

ISOM-130M-MoE demonstrates System-2 deliberative reasoning on ARC and symbolic math benchmarks, punching significantly above its 134.89M parameter weight class.

ARC-Challenge (Abstract Reasoning Corpus)

Speculative Decoding Role

ISOM-130M-MoE is designed as the draft model for ISOM-Qwen-1.5B-Instruct and ISOM-Qwen2.5-Coder-1.5B-Instruct in speculative decoding pipelines:

# Conceptual speculative decoding setup
from transformers import pipeline

draft_model = "Prannesshkva/ISOM-130M-MoE"        # fast drafter
target_model = "Prannesshkva/ISOM-Qwen2.5-Coder-1.5B-Instruct"  # verifier

# Result: 2x-3x throughput increase with identical output quality

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Prannesshkva/ISOM-130M-MoE"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

prompt = "Solve step by step: If a train travels 60 km/h for 2.5 hours, how far does it travel?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

IP & Governance

The ISOM architecture (Isometric State Operator Manifold), its recurrent memory compression operators, bounded-state SSM layers, and deliberative reasoning integration are the intellectual property of Prannesshkva (Prannesh K. V. A.).

• Contact & Research Collaboration: LinkedIn — Prannesh K. V. A.
• Zenodo DOI: 10.5281/zenodo.22649142

The base Mamba SSM layer implementation follows the Mamba project (Apache 2.0). This model and all ISOM derivatives are released under Apache 2.0.