Oğuz-MoE-354M-Base-v2

Continued-healing checkpoint of Oğuz-MoE-354M-Base (v1): +150M tokens of MoE healing (300M cumulative MoE tokens, ~150M routed tokens per expert), gradient checkpointing re-enabled (non-reentrant), and a new 2048-context training phase. 354.2M total parameters, ~205.5M active per token (top-2 of 4 experts).

Like v1, this is a base (pretrained-only) checkpoint: it completes text, it is not instruction-tuned.

⚠️ This model uses a custom architecture (MoeLlamaForCausalLM). Load it with trust_remote_code=True.

Model Summary

What changed vs. v1

Expert Divergence Analysis

Pairwise cosine between expert weight matrices, and relative Frobenius drift of the inherited 1536-dim subspace versus the original dense FFN (the upcycling source), measured on the released checkpoints:

The in-run monitor (pairwise cosine of down_proj on layers 0/10/20, every 500 optimizer steps) declined monotonically 0.8803 → 0.8736 and plateaued over the final ~10k steps. Interpretation: doubling the per-expert token budget yields continued but strongly diminishing orthogonalization — at this scale, top-2-of-4 upcycled experts remain partially specialized near-ensembles rather than fully distinct domain/syntactic pathways. Divergence concentrates in down_proj/up_proj and middle layers (max per-expert drift 0.49 at layer 6); gate projections stay most similar (0.91–0.94).

Full per-layer reports: divergence_baseline.json and divergence_final.json in this repository.

Training Pipeline (Lineage)

Total token exposure across all stages: ~2.35B

Datasets & Tokenizer

• Tokenizer: ytu-ce-cosmos/turkish-gpt2 (50,257 tokens).
• Corpus: streamed interleaved mixture (seed 4242361): 30% Cosmos-Turkish-Corpus-v1.0, 70% FineWeb-2 (tur_Latn).
• Tokenized offline into a flat uint16 binary (1.6B tokens), memory-mapped during all stages.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "Yigit-Karaman/Oguz-MoE-354M-Base-V2"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,      # required: custom MoE architecture
        dtype=torch.bfloat16,          # was: torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Türkiye'nin başkenti"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Training Monitoring (Stage 4, v2 run)

• VRAM preflight (real optimizer step per phase): 3.56 GB (8×512), 4.30 GB (4×1024), 4.30 GB (2×2048) on an 8 GB RTX 4060 — no system-memory offloading.
• Resume robustness: the run survived 3 interruptions (including Ctrl+C mid-backward); every resume restored model weights, 8-bit AdamW state, scheduler, and token offset correctly.
• Expert load balance (EMA): 23–26% per expert throughout; no collapse at any point.
• Aux loss: 1.00–1.04; mildly elevated (1.02–1.04) during and after the first-ever 2048-context phase — consistent with the router adapting to longer-range dependencies, and still well inside the healthy band.
• LM loss: per-microbatch fluctuations ~2–5 on heterogeneous web text; occasional near-zero microbatches correspond to repetitive content (memorization), not instability.
• Learning rate: cosine 1e-4 → ~7e-11 over 9,155 optimizer steps (36,620 microsteps).
• Throughput: 6,000–8,300 tok/s with gradient checkpointing enabled.
• Routed tokens per expert in Stage 4: ~75M (cumulative ~150M including v1).

Reproducibility

• Global seed: 4242361 (Python / NumPy / PyTorch / CUDA, and dataset interleaving).
• Hardware: 1× NVIDIA GeForce RTX 4060 (8 GB), Windows (WDDM).
• Software: PyTorch (bf16 autocast, SDPA), Transformers custom-code loading, bitsandbytes 0.50.2 (AdamW8bit), Safetensors.
• Data pipeline: offline uint16 token binary (1.6B tokens), memory-mapped streaming with tail+head wraparound.
• Optimizer (Stage 4): AdamW8bit, lr 1e-4, wd 0.1, cosine over 9,155 steps; gradient accumulation 4; 16,384 tokens per update in all three context phases; grad-norm clip 1.0.
• Checkpointing: time-based (30 min), saved only immediately after optimizer steps; loaded with map_location="cpu" and bitsandbytes move_to_device.
• MoE specifics: 4 experts / top-2; expert intermediate 2048; router Linear(576→4); softmax-normalized top-2 gates; Switch-style aux loss coefficient 0.01; gradient checkpointing use_reentrant=False (keeps the aux-loss autograd graph intact).

Limitations & Biases

• Turkish (Latin script) only; performance on other languages or non-Latin Turkish scripts is expected to be poor.
• Web-text biases: inherits stereotypes, errors, and topical biases from public Turkish web corpora.
• Memorization: occasional near-zero-loss microbatches indicate some rote memorization of repetitive web content.
• Small model: prone to factual errors, hallucination, and weak long-range reasoning.
• Base model: not instruction-tuned or aligned; unsuitable for conversational use without further fine-tuning.
• Partial expert specialization: experts behave as a softly-routed ensemble, not cleanly separated domain modules (see Divergence Analysis).
• Custom code: loading requires trust_remote_code=True; review modeling_moe_llama.py before execution.

Intended Use

Research and prototyping for Turkish NLP: language-modeling experiments, fine-tuning/adapter studies, and as a worked example of progressive depth expansion + MoE upcycling + continued healing on consumer hardware. Not intended for production deployment or safety-critical applications.

License

Apache 2.0 (see the LICENSE file in this repository). Upstream attributions:

• FineWeb-2, licensed ODC-BY 1.0.
• Cosmos-Turkish-Corpus-v1.0, courtesy of the YTÜ-Cosmos project.
• Tokenizer and base architecture configuration inspired by ytu-ce-cosmos/turkish-gpt2.

Citation

@misc{Oguz-MoE-354M-Base-V2,
  title        = {Oğuz-MoE-354M-Base-v2: Continued-Healing Checkpoint of a Turkish Mixture-of-Experts Language Model},
  author       = {Yiğit Karaman},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Yigit-Karaman/Oguz-MoE-354M-Base-V2}}
}

Acknowledgements

Tokenizer and part of the training corpus courtesy of the YTÜ-Cosmos project; Turkish web text from HuggingFace FineWeb-2. Depth-upscaling and MoE upcycling techniques follow the public literature on layer duplication and expert upcycling of dense checkpoints. Built with Hugging Face Transformers, bitsandbytes, and PyTorch.