🧪 Vigyan 1.5B 4× MoE (Sparse Mixture of Experts)

Classification: Experimental Research Artifact / Compact Edge MoE
Architecture Target: 4-Expert Sparse MoE with Top-1 Active Routing
Base Model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
Research Lab: Vigyan AI Sovereign Initiative

📌 1. Executive Summary & Research Context

Vigyan-1.5B 4× MoE is an experimental Sparse Mixture of Experts model upcycled from 4 specialized domain LoRA adapters (Science, Technology, Engineering, Mathematics).

It was assembled to evaluate whether post-hoc expert stitching on compact language models ($\le 3\text{B}$) can deliver multi-domain specialization at single-expert inference latency with a sub-1GB RAM footprint.

A pre-quantized standalone GGUF (vigyan-1.5b-4x-moe.Q4_K_M.gguf) is available directly within this repository for instantaneous local execution on CPU or mobile.

🚀 1-Click Interactive Google Colab Demo

Test this model on Google Colab with an embedded Gradio chat interface:

📐 2. Architectural Blueprint

• Base Foundation: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
• Active Routing: Top-1 active expert per token + shared residual STEM anchor.
• Assembled Experts:
  • shreyansh12183/vigyan-1.5b-adapter-science
  • shreyansh12183/vigyan-1.5b-adapter-technology
  • shreyansh12183/vigyan-1.5b-adapter-engineering
  • shreyansh12183/vigyan-1.5b-adapter-mathematics
• Gating Mechanism: gate_mode: hidden with contrasting negative prompt calibration.

📊 3. Empirical Benchmark Observations

• Entropy Score: Observed gating logit entropy $\approx 1.1152$, demonstrating router domain sensitivity.
• Inference Velocity: Evaluated at $\approx 16-22\text{ tokens/second}$ on consumer AMD Ryzen 5 CPU.
• Failure Modes Identified:
  1. Delimiter Blindness: Without general conversational anchor replay during adapter training, the model degrades into token echo loops when receiving general chat greetings.
  2. Attractor Collapse: Due to uncoordinated expert state representations, consecutive tokens routed across disparate experts trigger representation collapse.

🔮 4. Future Roadmap

Our primary production focus has transitioned to Allen AI's native OLMoE-1B-7B (64 natively pretrained experts with auxiliary load-balancing loss). This checkpoint serves as foundational reference data for the Post-Hoc Upcycling Fragility Theorem.

📜 License & Sovereign Attribution

Released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

✅ Permitted Free Use:

• Academic research, university coursework, student study, and personal experimentation.

🚫 Commercial Restrictions:

• Any commercial deployment, paid SaaS API, commercial tutoring platform, or enterprise wrapper requires an explicit commercial license.
• Founder & Chief Architect: Shreyansh Singh
• Organization: Vigyan AI / ExperimentLab.in

📜 Academic Research Paper & Citation

This architecture is formally documented and validated in the Vigyan AI Technical Paper Series (Vol 01):

• Paper Title: Sparse Mixture-of-Experts (MoE) Upcycling and Router Stabilization
• Author: Shreyansh Singh (Founder & Lead Researcher, Vigyan AI / ExperimentLab.in)
• Dataset & LaTeX Sources: 🤗 HuggingFace Paper Hub
• Interactive Kaggle Dataset: 📦 Kaggle Paper
• GitHub Repository: 🐙 vigyan-ai-journey-docs

@article{singh2026moe},
  title   = {Sparse Mixture-of-Experts (MoE) Upcycling and Router Stabilization},
  author  = {Singh, Shreyansh},
  journal = {Vigyan AI Technical Whitepaper Series},
  year    = {2026},
  publisher = {ExperimentLab.in},
  url     = {https://huggingface.co/datasets/shreyansh12183/vigyan-ai-research-papers}
}