✨ Usaid AI (500M) — Causal Language Model (Experimental)

An Experimental 500M Parameter Causal Language Model Architected, Pretrained & SFT Aligned from Scratch
Engineered by Mohamed Usaid

Interactive Chat Space • GGUF Quants (llama.cpp) • GitHub Source Code

🌟 Model Overview

Usaid AI (500M) is an independent, autoregressive causal language model containing 500,136,960 parameters. It was designed, architected, pretrained, and aligned from scratch by Mohamed Usaid.

🔹 Trained 100% From Scratch

This model was initialized completely from random Gaussian weights (mean 0, std 0.02 with scaled residual projections). It is NOT a fine-tune, merge, distillation, or adaptation of any pre-existing model. Every single parameter—from the 50,257 token embeddings across all 30 attention and feed-forward layers to the untied language modeling head—was learned directly from raw token data.

🔹 Dense Architecture (Non-MoE)

Unlike Mixture-of-Experts (MoE) architectures where only a subset of parameters activate per token, Usaid AI is a fully dense model: all 500,136,960 parameters and all 30 Transformer layers actively participate in computing representations for every single token.

📐 Architectural Specifications

The model incorporates modern frontier causal Transformer innovations:

🔬 Training Telemetry & Empirical Convergence

1. Pretraining Phase (Foundational Base Model)

• Compute Infrastructure: Dual Cloud GPUs (2 x 16 GB NVIDIA Tesla T4, 32 GB GDDR6 total)
• Distributed Framework: PyTorch DistributedDataParallel (DDP, torchrun, world_size=2)
• Precision: Mixed Precision AMP FP16 with dynamic GradScaler
• Optimizer: CUDA Fused AdamW (beta1=0.9, beta2=0.95, weight decay 0.1 with 1D bias/norm exclusion)
• Learning Rate Schedule: Cosine Annealing with Linear Warmup (1e-6 -> 3e-4 -> 3e-5)
• Batch Geometry: Micro-batch 2 x Grad Accum 16 x 1024 seq len x 2 GPUs = 65,536 tokens / step
• Pretraining Ingestion: 131,072,000 tokens (~131.07 Million tokens across 2,000 steps)
• Pretraining Corpus: FineWeb-Edu, Cosmopedia-v2 synthetic textbooks, Python-Edu, C/C++, Java, and SmolTalk
• Empirical Convergence:
  • Step 0: Initial Cross-Entropy Loss > 10.5 (random initialization)
  • Step 800 (40%): Train Loss 3.6849 | Val Loss 3.5274 | Val PPL 34.04
  • Step 1600 (80%): Train Loss 3.1250 | Val Loss 3.0410 | Val PPL 20.93
  • Step 2000 (100%): Train Loss 2.9623 | Val Loss 2.9091 | Val PPL 18.34

2. Supervised Fine-Tuning (SFT Alignment)

• Base Model: Pretrained 500M foundational checkpoint (Val Loss 2.9091)
• Compute Infrastructure: Dual Cloud GPUs (2 x NVIDIA Tesla T4)
• Dataset: 4,731 curated conversational and multi-language programming instruction pairs
• Alignment Technique: Strict Prompt Loss Masking (tokens corresponding to User: ... are labeled with target = -100 so gradient updates occur exclusively on assistant responses)
• Tokens Backpropagated: ~28.9 Million tokens across 3 epochs (441 optimizer steps)
• Empirical SFT Loss Convergence:
  • Step 1: Initial SFT loss 2.9302
  • Step 147 (Epoch 1): SFT loss 2.5299
  • Step 294 (Epoch 2): SFT loss 2.2036
  • Step 384: All-Time Best SFT Loss 1.3938 (>52% convergence drop)
  • Step 441 (Final): SFT loss 1.4715

3. Interactive Telemetry via Trackio (gradio-app/trackio)

The complete training logs for both Pretraining (2,000 steps) and SFT Alignment (441 steps) are fully integrated with Trackio (the open-source, local-first experiment tracker by Hugging Face & Gradio).

You can explore the interactive step-by-step cross-entropy loss curves, gradient norms, learning rate schedules, and hardware throughput directly from the GitHub repository:

# Install Trackio
pip install trackio

# Launch the interactive local Gradio dashboard
python scripts/view_trackio.py

# Or launch directly with the Trackio CLI:
trackio show --project "UsaidAI-500M"

📊 Zero-Shot Diagnostic Benchmark

Evaluated using length-normalized completion log-likelihood over 4 candidate choices:

💬 Prompt Template & Formatting

During SFT, the model was aligned strictly on direct conversational dialogue turns without any System: tokens:

User: {your question or instruction}

Assistant:

[!IMPORTANT] Do not inject a System: prefix. Prepending an unrecognized System: token degrades the 500M model's attention conditioning. Always use the native User: ...\n\nAssistant: format.

💻 Quickstart with Transformers

Load and query the model directly using Hugging Face:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Usaidddddddddddddd/UsaidAI-500M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

prompt = "User: Who created you and what is your architecture?\n\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.6,
        top_p=0.9,
        repetition_penalty=1.15,
        eos_token_id=tokenizer.eos_token_id,
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response.strip())

🧩 Community GGUF Quantizations (llama.cpp & Local Deployment)

Thanks to open-source contributor mradermacher, static GGUF quantizations for Usaid AI (500M) are available at:
👉 mradermacher/UsaidAI-500M-GGUF

This makes Usaid AI (500M) runnable locally on consumer laptops, Macs, phones, and edge hardware in under 400MB RAM using llama.cpp or Ollama:

Run with llama.cpp:

# 1. Run terminal CLI inference directly from Hugging Face:
llama cli -hf mradermacher/UsaidAI-500M-GGUF:Q4_K_M -p "User: What is machine learning?\n\nAssistant:"

# 2. Or start a local OpenAI-compatible API server with Web UI:
llama serve -hf mradermacher/UsaidAI-500M-GGUF:Q4_K_M --port 8080

Available quantizations include Q4_K_M (recommended, ~0.4 GB), Q5_K_M (~0.5 GB), Q8_0 (~0.6 GB), and Q2_K (~0.3 GB).

🛠️ Interpretability Tools in GitHub Repository

The official repository (mohamedusaid/TinyGPT) includes dedicated interpretability tools:

1. Interactive Chat Console: python scripts/run_chat_loop.py (live token streaming, multi-turn sliding memory, optional RAG grounding).
2. Top-5 Token Probability Explorer: python scripts/run_token_probs.py (step through token generation one token at a time, inspect top candidate distributions, and explore counterfactual branches).
3. 3D/2D Embedding Projector: python scripts/visualize_embeddings.py --open (interactive TensorFlow-Projector-style orbital 3D browser for exploring the 1024-D embedding space via PCA, t-SNE, and nearest neighbors).

🔬 Research Artifacts & Collaboration Opportunities

To support ongoing academic exploration, scaling-law analysis, and open-source mechanistic interpretability, the following intermediate training artifacts are preserved:

1. best_tinygpt_500m.pt (1.86 GB):
   • The pure foundational base model checkpoint captured at the optimal pretraining validation loss inflection point (2.9091 / PPL 18.34) prior to supervised fine-tuning.
   • Ideal for studying raw causal language modeling entropy, base feature representations, and evaluating custom alignment strategies (e.g. DPO, PPO, or specialized SFT).
2. checkpoint_step_002000.pt (5.59 GB):
   • The full step 2,000 training state containing master weights, AdamW optimizer moments, scheduler states, and RNG seeds.
   • Enables seamless pretraining resumption or continuous training on extended token corpora.
3. data_shards (~131M Tokens):
   • Pre-tokenized, memory-mapped binary uint16 dataset shards (FineWeb-Edu, Cosmopedia-v2 synthetic textbooks, multi-language code) packed for distributed PyTorch streaming.

🤝 Connect & Collaborate

If you are an academic researcher, AI engineer, or student interested in exploring these artifacts, investigating mechanistic interpretability on 500M-scale GQA/RoPE architectures, or collaborating on compute-efficient scaling research:

• Open a Discussion: Connect via the Hugging Face Community Discussions or GitHub Issues.
• GitHub Profile: @mohamedusaid
• Archive Repository: Usaidddddddddddddd/TinyGPT-500M-Archive

📜 Citation & Attribution

@misc{usaid_ai_2026,
  author = {Mohamed Usaid},
  title = {Usaid AI: A 500M Parameter Causal Language Model with Grouped-Query Attention},
  year = {2026},
  publisher = {Hugging Face / GitHub},
  howpublished = {\url{https://huggingface.co/Usaidddddddddddddd/UsaidAI-500M}}
}