⚛ Shreyansh-STEM-AI-2B-v3: 22-Layer Depth-Upscaled (DUS) Foundation Model

Shreyansh-STEM-AI-2B-v3 is a sovereign compact foundation model for scientific, physical, and mathematical derivation, engineered through SOLAR-style Depth Up-Scaling (DUS) and Continual Pre-Training (CPT) Seam Healing.

🔬 Architectural Mechanics: The DUS & CPT Healing Journey

[Standard Base SLM]
        │
        ▼  SOLAR-Style Block Splicing
[22-Layer DUS Architecture]  -->  (Experienced Interface Discontinuity / Layer Seam Shock)
        │
        ▼  Continual Pre-Training (CPT) across 2,400+ Parquet Shards
[CPT-Healed Manifold: shreyansh-1B-SLM-pretrain-stem-english]
        │
        ▼  All-Module Linear SFT with Dynamic Early-Stopping (Loss 0.1335)
[Cured 0.048 Overfit Collapse --> Sovereign 2B STEM Foundation]

1. Depth Up-Scaling (DUS)

To surpass standard 2B parameter capacity without requiring training from scratch, intermediate transformer layers were duplicated and spliced, expanding the model depth to 22 layers with a hidden dimension of $d=2048$.

2. Continual Pre-Training (CPT) Seam Healing

Splicing transformer blocks introduces interface discontinuity along the residual stream. To heal these seams, the model underwent Continual Pre-Training (CPT) over the multi-gigabyte shreyansh-1B-SLM-pretrain-stem-english corpus (over 2,400 parquet shards). This enabled the spliced attention heads to smoothly adapt their query-key projection subspaces.

3. Averting the Historical 0.048 Overfit Collapse

Earlier fine-tuning iterations succumbed to a catastrophic 0.048 loss collapse (memorized lookup table). By implementing All-Module Linear Targeting across all 7 projections: $$\text{Modules} = [\text{q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj}]$$ combined with Dynamic Early Stopping at loss $\approx 0.1335$ at step 60, representation collapse was eliminated, achieving a +162.5% relative accuracy improvement on empirical multi-step benchmarks when coupled with deterministic AST tools.

🚀 1-Click Interactive Google Colab Demo

Test this model instantly on Google Colab with an embedded Gradio chat interface:

💻 Running Locally via Ollama & Transformers

1. Via Hugging Face Transformers (4-bit NF4)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL_ID = "shreyansh12183/Shreyansh-STEM-AI-2B-v3"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto"
)

prompt = "Derive the escape velocity from the surface of Earth using energy conservation."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Standalone GGUF on CPU via Ollama

ollama run hf.co/shreyansh12183/Vigyan-Models-GGUF:vigyan-2b-stem-q4_k_m.gguf

📜 License & Sovereign Attribution

Released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

✅ Permitted Free Use:

• University coursework, academic research, high-school exam study (JEE, CBSE), and personal non-commercial exploration.

🚫 Commercial Restrictions:

• Any commercial deployment, paid SaaS API, commercial tutoring platform, or enterprise wrapper requires an explicit commercial license.
• Founder & Chief Architect: Shreyansh Singh
• Organization: Vigyan AI / ExperimentLab.in

📜 Academic Research Paper & Citation

This architecture is formally documented and validated in the Vigyan AI Technical Paper Series (Vol 02):

• Paper Title: Depth Up-Scaling (DUS) and Continual Pre-Training Seam Healing
• Author: Shreyansh Singh (Founder & Lead Researcher, Vigyan AI / ExperimentLab.in)
• Dataset & LaTeX Sources: 🤗 HuggingFace Paper Hub
• Interactive Kaggle Dataset: 📦 Kaggle Paper
• GitHub Repository: 🐙 vigyan-ai-journey-docs

@article{singh2026dus},
  title   = {Depth Up-Scaling (DUS) and Continual Pre-Training Seam Healing},
  author  = {Singh, Shreyansh},
  journal = {Vigyan AI Technical Whitepaper Series},
  year    = {2026},
  publisher = {ExperimentLab.in},
  url     = {https://huggingface.co/datasets/shreyansh12183/vigyan-ai-research-papers}
}