ISOM-R1-Edge-130M-MoE (Upgraded with ISOM-R2 1M Engine)

0.0469 MB Invariant Recurrent State • ISOM-R2 1,000,000-Token Retrieval Engine • Verified on NVIDIA Tesla T4

🚀 ISOM-R2 1,000,000-Token Engine Upgrade

This repository has been upgraded with the ISOM-R2 architecture (ISOMR2VirtualSVDCache + ISOMR2Engine), the same 1M-token streaming-retrieval engine powering Prannesshkva/Aazhi-Coder-1.5B (verified 1,000,000-token benchmark notebook):

• Combines ISOM-R1-Edge-130M-MoE's 0.0469 MB continuous quasi-unitary SSM recurrence with ISOM-R2's Hierarchical Paged Virtual SVD Cache (ISOMR2VirtualSVDCache) and Hybrid Lexical + Dense Micro-Window Retrieval.
• Automatically activates chunked streaming + salient micro-window retrieval inside model.generate() for contexts $> 2,048$ tokens (use_isom_r2_svd=True enabled by default in config.json), or via ISOMR2Engine in isom_r2_engine.py.

Overview

ISOM-R1-Edge-130M-MoE is an ultra-compact 134.89-million parameter (58.27M active per token) continuous recurrent state-space architecture. Engineered specifically as a high-speed speculative decoding drafter and edge deliberation engine, it processes arbitrarily long token streams without materializing quadratic attention matrices.

📊 Audited Empirical Hardware Telemetry (NVIDIA Tesla T4, Kaggle Cloud)

Evaluated on an NVIDIA Tesla T4 (14.56 GB / 14,911.7 MB total VRAM, PyTorch 2.10.0+cu128, CUDA 12.8, Kaggle Cloud) across an authentic, unpadded continuous literature stream (Pride and Prejudice, 728,846 characters) from 10,000 up to 100,000 continuous tokens:

Continuous Recurrent Scaling (10,000 to 100,000 Tokens)

Key Architectural Verification: Across the entire 100,000 continuous token stream, the recurrent hidden state footprint remains strictly invariant at 0.0469 MB (48 KB). The continuous Cayley SO(d) manifold preserves numerical isometry, delivering processing speeds scaling up to 21,049.0 tokens/second on a single Tesla T4 GPU.

Model Specifications

Quickstart Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Prannesshkva/ISOM-R1-Edge-130M-MoE-Beta-Prototype"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = "Solve step by step: If a car travels 90 km/h for 3.5 hours, what is the total distance traveled?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        temperature=0.7,
        do_sample=True
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Optional: Hybrid Neuro-Symbolic Agent Pipeline
from modeling_isom_130m_moe import ISOMDeliberativeAgent
agent = ISOMDeliberativeAgent(model=model, tokenizer=tokenizer)
result = agent.solve("Solve step by step: A train travels 60 km in 45 minutes. What is its speed in km/h?")
print("Agent Trace:", result.get("thought_trace"))
print("Final Answer:", result.get("final_answer"))

⚠️ Training Notice: MoE Load-Balancing

ISOMStaticMoE uses softmax top-2 routing with weight renormalization but does not include an auxiliary load-balancing loss. During inference this has no effect — the model routes normally. During fine-tuning, the absence of a balancing term means the router is free to concentrate all tokens on a small subset of experts (expert collapse), leaving the remaining experts with near-zero gradient signal and wasted capacity.

Mitigation for fine-tuning:
Add a Switch Transformer-style auxiliary loss to your training loop before calling loss.backward():

# After computing CE loss, add auxiliary load-balancing loss
router_logits = ...  # collect router_logits from each ISOMStaticMoE layer
probs = torch.softmax(router_logits, dim=-1)                        # [T, E]
f_i = probs.mean(dim=0)                                             # expert fraction
P_i = (probs > 0).float().mean(dim=0)                              # expert dispatch rate
aux_loss = config.num_experts * (f_i * P_i).sum()
total_loss = ce_loss + 0.01 * aux_loss

Without this, or an equivalent expert-parallelism regularizer, training runs longer than ~1,000 steps risk expert utilization collapse to 2 of 8 experts.

Citation & Licensing

@software{isom_edge_130m_2026,
  author = {Prannessh K.V.A.},
  title = {ISOM-R1-Edge-130M-MoE: Continuous Recurrent SSM + Mixture-of-Experts Drafter},
  year = {2026},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.22649142},
  url = {https://doi.org/10.5281/zenodo.22649142}
}

• Sole Author & Architect: Prannessh K.V.A.
• LinkedIn: Prannessh K.V.A.
• License: Governed by CC BY-NC-ND 4.0 (Non-Commercial Research) & Enterprise Commercial Terms. See LICENSE.

Notice of Non-Endorsement & Independent Lineage

[!IMPORTANT] Independent Architecture Work: ISOM-R1-Edge-130M-MoE is an original standalone research architecture engineered solely by Prannessh K.V.A. (Author, Architect & IP Holder). It implements continuous isometric state operator manifolds (Cayley SO(d)) combined with 8 SwiGLU Mixture-of-Experts feedforward layers. Governed by CC BY-NC-ND 4.0 & Enterprise Commercial Terms (see LICENSE).