🎼 ORCH Next.js 3B
A 3 billion parameter decoder-only transformer trained from scratch for generating complete Next.js applications — pages, API routes, Prisma schemas, Tailwind components, configs.
TL;DR
| Parameters | ~3.0 Billion |
| Architecture | Custom LLaMA-style decoder-only transformer |
| Training | From scratch — no base model |
| Vocabulary | 32,000 (custom) |
| Context length | 16,384 tokens |
| Hardware | NVIDIA A40 48GB (RunPod) |
| Training duration | ~2 hours (3 epochs, ~29,000 steps) |
| License | Apache 2.0 |
What this is
The largest model in the from-scratch ORCH lineup. Designed for full-stack Next.js generation: not just snippets, but complete project structures — TypeScript components, App Router pages, server actions, Prisma schemas, Tailwind utilities, and configuration files.
This is not a fine-tune of any pretrained model. Architecture and weights are trained end-to-end on curated Next.js repositories.
Architecture
Layers: 32
Hidden size: 2,560
Intermediate size: 10,240
Attention heads: 32
KV heads (GQA): 8
Max position: 16,384
RoPE theta: 10,000
Activation: SwiGLU
Normalization: RMSNorm
Tied embeddings: no
Vocab size: 32,000 (custom)
Standard LLaMA-style ingredients (RoPE, GQA, SwiGLU, RMSNorm) at scale. The 16,384-token context length allows the model to keep a meaningful portion of a project in context during generation.
Training
- Data: curated Next.js repositories from GitHub
- Hardware: single NVIDIA A40 48GB (RunPod)
- Duration: ~2 hours
- Epochs: 3 (~29,000 steps)
- Batch size: 1 with gradient accumulation = 16
- Sequence length: 512 (training) — the 16K context length applies at inference
- Precision: BFloat16
Generated artifacts
The model is trained to produce complete project structures:
- Frontend: React components, App Router pages, layouts, hooks (TypeScript)
- Backend: API routes, server actions, middleware
- Database: Prisma schemas, query utilities
- Styling: Tailwind CSS, shadcn/ui-style component patterns
- Configuration:
package.json,tsconfig.json,next.config.js
Usage
Custom PyTorch format — use the ORCH inference code (not yet published):
import torch
from orch.model.config import OrchConfig
from orch.model.transformer import OrchForCausalLM
model = OrchForCausalLM.from_pretrained("raihan-js/orch-nextjs-3b")
# ... use the same tokenizer.json from the repo
Intended use
- Full Next.js project bootstrapping from a natural language description
- Research into scaling SLMs trained from scratch on domain-specific data
- A from-scratch baseline to compare against fine-tuned models like ORCH-7B
Limitations
- Data-limited and undertrained: trained once on a rented A40 with a limited corpus (not compute-optimal), never benchmarked and never deployed. Treat output as research-grade, not production-grade.
- Training data scale: 3 epochs on curated Next.js repos. Don't expect the world knowledge of a 7B+ general-purpose model.
- Sequence length during training (512): capable of using long context at inference but may show degradation outside the training distribution.
- No safety alignment.
- Custom format: requires the ORCH inference code, not loadable with
AutoModelForCausalLM.
Related models
raihan-js/orch-fusion— 272M sibling, tiny 2,103 vocabraihan-js/orch-nextjs-350m-v2— 287M sibling, 16k vocabraihan-js/orch-7b— alternate approach: QLoRA fine-tune of DeepSeek Coder 6.7B- ORCH Studio — Gradio demo Space (currently runs ORCH-7B)
Author
Akteruzzaman Raihan Sikder — AI/ML engineer. Founding engineer and AI/ML lead at VETR Proposal; previously CTO of ClarioScope AI (2024–2026, sunset). Portfolio · GitHub.
Citation
@misc{sikder2025orchnextjs3b,
title = {ORCH Next.js 3B: A 3-Billion-Parameter Decoder-Only Transformer Trained From Scratch for Full-Stack Next.js Code Generation},
author = {Sikder, Akteruzzaman Raihan},
year = {2025},
url = {https://huggingface.co/raihan-js/orch-nextjs-3b}
}