project-828-gpt-base
A 398.7M parameter Mixture-of-Experts (MoE) causal language model optimized for code generation and technical reasoning, pretrained on ~60B tokens.
398.7M total parameters, ~286M active per token — 4 routed experts + 1 shared expert with top-2 routing per token.
Model Architecture
| Property | Value |
|---|
| Architecture | Decoder-only Transformer with MoE FFN layers |
| Total Parameters | 398.7M |
| Active Parameters / Token | ~286M |
| Hidden Dimension ($d_{\text{model}}$) | 768 |
| Intermediate Size ($d_{\text{ff}}$) | 760 |
| Hidden Layers | 24 |
| Attention Heads / KV Heads | 12 / 6 (GQA 2:1) |
| Head Dimension | 64 |
| Routed Experts | 4 |
| Active Experts / Token | 2 (Top-2) |
| Shared Experts | 1 |
| Context Length | 2048 (extensible to 8192 via YaRN) |
| Vocabulary Size | 49152 |
| Precision | BFloat16 Mixed Precision |
| Positional Encoding | RoPE with YaRN scaling support |
Parameter Breakdown
| Component | Parameters |
|---|
| Embeddings | ~37.7M |
| Unembedding | ~37.8M |
| Attention (×24 layers) | ~1.8M each |
| MoE FFN (×24 layers) | ~11.7M each |
| Layer Norms + Misc | ~0.07M |
| Total | 398.7M |
Key Architectural Features
- Grouped Query Attention (GQA): 12 query heads with 6 key-value heads (2:1 ratio) for memory-efficient attention
- QK-Norm: RMSNorm applied to query and key projections before RoPE for attention stability
- Auxiliary-Loss-Free MoE Routing: Sigmoid gating with dynamic bias adjustment (DeepSeek-V3 paper) — achieves near-perfect ~25% utilization per expert without auxiliary losses
- SwiGLU Activation with Soft-Clamping:
limit=7.0 prevents activation explosions during long training runs
- Batched Expert Dispatch: Sort-and-slice dispatch with
searchsorted boundaries for contiguous memory access
- RoPE with YaRN Extension: Base context of 2048 tokens, extensible to 8192 via YaRN scaling
Training Details
Phase 1 — Pretraining (~60B tokens)
| Training Config | Value |
|---|
| Hardware | H200 GPU |
| Peak Learning Rate | 3e-4 |
| Min Learning Rate | 3e-5 |
| Scheduler | WSD (Warmup-Stable-Decay) |
| Warmup Steps | 500 |
| Total Steps | 101,726 |
| Effective Batch Size | 37 × 8 = 296 sequences |
| Tokens per Step | ~0.61M |
| Gradient Clipping | 1.0 |
Training Data Mix
| Dataset | Weight | Category |
|---|
starcoderdata — Python | 14 | Source Code |
starcoderdata — JavaScript | 8 | Source Code |
starcoderdata — Java | 6 | Source Code |
starcoderdata — TypeScript | 4 | Source Code |
starcoderdata — C++ | 6 | Source Code |
starcoderdata — C | 4 | Source Code |
starcoderdata — C# | 3 | Source Code |
starcoderdata — Go | 4 | Source Code |
starcoderdata — Rust | 3 | Source Code |
starcoderdata — PHP | 3 | Source Code |
fineweb-edu-dedup | 20 | General Knowledge |
cosmopedia-v2 | 7 | General Knowledge |
wikipedia-en | 3 | General Knowledge |
finemath-4plus | 8 | Math / Reasoning |
stackexchange (programming/CS) | 7 | CS / Engineering |
Category breakdown: Source Code 55% · General Knowledge 30% · Math/Reasoning 8% · CS/Engineering 7%
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "AkshithAI/project-828-gpt-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype="bfloat16",
device_map="auto",
trust_remote_code=True
)
prompt = "def binary_search(arr, target):"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=200,
temperature=0.7,
top_p=0.9,
do_sample=True,
use_cache=False,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
@misc{project828,
author = {AkshithAI},
title = {Project 828: MoE Transformer with Training Pipeline},
year = {2025},
publisher = {GitHub},
url = {https://github.com/AkshithAI/project_828}
}