🧠 AlterEgo-373M

A 373-million-parameter language model designed, trained, and served entirely from scratch.

Introduction

AlterEgo is a small, decoder-only language model built from the ground up - not a fine-tune of an existing model. Every part was written from zero: the transformer architecture, the training loop, the tokenizer wiring, and the KV-cached inference engine. It was pre-trained on ~10B tokens of high-quality educational web text and then instruction-tuned for chat.

It is the model at the heart of LLME, a self-hosted, end-to-end-encrypted LLM platform (think LM Studio / Open WebUI / Ollama, also built from scratch). LLME can serve AlterEgo alongside llama.cpp GGUF models and the Gemini API; AlterEgo is the "house" model it was designed around.

This repository contains the model. The training and architecture code lives in the AlterEgo repo; the serving platform lives in the LLME repo.

Two formats are published. This repo is the Hugging Face LlamaForCausalLM conversion, for drop-in use with transformers, vLLM, and GGUF tooling. The original checkpoint - in AlterEgo's own from-scratch architecture, exactly as trained - is published separately as jbomdev/alterego_raw. This version is a numerically-lossless conversion of it (verified: max logit difference ~1e-6).

What it is and isn't. AlterEgo is a research / learning artifact - a demonstration of the full modern LLM pipeline (architecture → pretraining → SFT → serving) at a scale one person can train on a single GPU. It is not a production assistant and won't compete with billion-parameter models. See Limitations.

Architecture

A modern Llama-style decoder (and, thanks to that, it loads as a standard LlamaForCausalLM).

Training

AlterEgo was trained in two stages on a single NVIDIA RTX 4090.

Stage 1 - Pretraining

Pre-trained on FineWeb-Edu (HuggingFaceFW), a quality-filtered educational subset of CommonCrawl.

The grad-norm settling to ~0.26 and the smooth cosine-shaped loss indicate stable training with no divergence.

Stage 2 - Supervised fine-tuning

Instruction-tuned on UltraChat-200K (HuggingFaceH4), formatted as multi-turn ChatML.

Hyperparameters

Evaluation

Benchmarked with EleutherAI's lm-evaluation-harness (0-shot).

For a 373M model trained on ~10B tokens these are solid: clearly above chance on science and commonsense (SciQ, PIQA, BoolQ, ARC-easy, HellaSwag) and on next-word prediction (LAMBADA — perplexity 62.3), with the expected near-chance results on the hardest reasoning sets (ARC-challenge, WinoGrande).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("jbomdev/AlterEgo")
model = AutoModelForCausalLM.from_pretrained("jbomdev/AlterEgo", torch_dtype=torch.bfloat16)

messages = [
    {"role": "system", "content":
     "You are Alter Ego, a small AI built from scratch. You're casual and direct. "
     "You're not great with facts, math, or current events - when you don't know "
     "something, just say so. You're better at chatting than at answering questions."},
    {"role": "user", "content": "Tell me something interesting about the ocean."},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

out = model.generate(
    ids,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.1,
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Recommended generation settings

These are the defaults AlterEgo was tuned and served with in LLME:

Lower the temperature toward 0.3–0.5 for steadier, more focused replies; it stops on <|im_end|> or <|endoftext|>.

Chat format

AlterEgo uses ChatML:

<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{message}<|im_end|>
<|im_start|>assistant

Run it locally (GGUF)

Feel free to use my pre-made GGUF's and quants by visiting The GGUF's and quants page. Or running the model with ollama.

Also, Because it's standard Llama format, you can convert to GGUF for Ollama / LM Studio / llama.cpp yourself:

python llama.cpp/convert_hf_to_gguf.py ./AlterEgo --outfile alterego-f16.gguf --outtype f16

Limitations

AlterEgo is a 373M-parameter model trained on a modest token budget, and it behaves like one:

• Capability - it can be factually wrong, repeat itself, and lose coherence on long or complex prompts. By its own (default) system prompt, it is "better at chatting than at answering questions."
• Language - English only.
• Safety - it is not safety- or preference-tuned (no RLHF/DPO). It can produce incorrect, biased, or undesirable content and must not be deployed in user-facing settings without additional safeguards.
• Bias - it inherits biases from FineWeb-Edu (web text) and UltraChat.

License

Released under the Apache 2.0 license. Training data is governed by the respective licenses of FineWeb-Edu and UltraChat-200K.

Citation

@misc{alterego2026,
  title  = {AlterEgo: A 373M language model trained from scratch},
  author = {J-bom},
  year   = {2026},
  url    = {https://github.com/J-bom/AlterEgo}
}

Credits - datasets: FineWeb-Edu (HuggingFaceFW), UltraChat-200K (HuggingFaceH4). Architecture follows the modern Llama-style design (RoPE, GQA, SwiGLU, RMSNorm); implementation, training, and serving by the author.