🧪 Vigyan 3B 4× MoE (Sparse Mixture of Experts)

Classification: Experimental Research Artifact / Test Model
Architecture Target: 4-Expert Sparse MoE with Top-1 Active Routing
Base Model: Qwen/Qwen2.5-3B-Instruct
Research Lab: Vigyan AI Sovereign Initiative

📌 1. Executive Summary & Research Context

Vigyan-3B 4× MoE is an experimental Sparse Mixture of Experts model upcycled from 4 specialized domain LoRA adapters (Science, Technology, Engineering, Mathematics).

It was assembled to evaluate whether post-hoc expert stitching on compact language models ($\le 3\text{B}$) can deliver multi-domain specialization at single-expert inference latency.

[!WARNING] Experimental Research Artifact:
This model is released publicly for academic replication and transparency. Extensive local testing revealed that post-hoc MoE upcycling without native end-to-end router pretraining suffers from attention-FFN desynchronization and autoregressive attractor loops under diverse conversational prompts.

📐 2. Architectural Blueprint

• Base Foundation: Qwen/Qwen2.5-3B-Instruct
• Active Routing: Top-1 active expert per token + shared residual STEM anchor.
• Assembled Experts:
  • shreyansh12183/vigyan-3b-adapter-science
  • shreyansh12183/vigyan-3b-adapter-technology
  • shreyansh12183/vigyan-3b-adapter-engineering
  • shreyansh12183/vigyan-3b-adapter-mathematics
• Gating Mechanism: gate_mode: hidden with contrasting negative prompt calibration.

📊 3. Empirical Benchmark Observations

• Entropy Score: Observed gating logit entropy $\approx 1.1152$, demonstrating router domain sensitivity.
• Inference Velocity: Evaluated at $\approx 16-22\text{ tokens/second}$ on consumer AMD Ryzen 5 CPU.
• Failure Modes Identified:
  1. Delimiter Blindness: Without general conversational anchor replay during adapter training, the model degrades into token echo loops when receiving general chat greetings.
  2. Attractor Collapse: Due to uncoordinated expert state representations, consecutive tokens routed across disparate experts trigger representation collapse.

🔮 4. Future Roadmap

Our primary production focus has transitioned to Allen AI's native OLMoE-1B-7B (64 natively pretrained experts with auxiliary load-balancing loss). This checkpoint serves as foundational reference data for the Post-Hoc Upcycling Fragility Theorem.

📄 License & Attribution

Released under the Apache 2.0 License by Shreyansh (@shreyansh12183).