ExtraAI-75M
ExtraAI is a 75,012,480-parameter, English-only decoder Transformer trained from random initialization. Its 32k byte-level BPE tokenizer is also trained from scratch; no pretrained model or pretrained tokenizer is used.
Architecture: 11 layers, width 640, 10 attention heads, SwiGLU width 1728,
RMSNorm, RoPE, PyTorch scaled-dot-product causal attention, tied embeddings, and
a 1024-token context. Exported after optimizer step 4,000. Both the original
Safetensors weights and a llama.cpp-compatible ExtraAI-75M-F16.gguf are included.
Training data (77,520 train conversations):
- agentic-ptb/grok-data
- 11-47/god_agent_grok4.4_cot_traces_20k
- WithinUsAI/Grok_4.4_Distilled
The training code applies a lightweight English filter and exact-text deduplication, then performs causal pretraining followed by assistant-only supervised training. Dataset licenses and the suitability of distilled outputs for your intended use must be checked independently before redistribution or commercial use.
Inference
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("REKS1337/ExtraAI-75M")
sys.path.insert(0, path)
from extraai_train import load_for_inference, generate_text
model, tokenizer = load_for_inference(path, device="cuda")
print(generate_text(model, tokenizer, "Explain why the sky is blue."))
This small model was trained under a strict single-session compute budget. Treat its answers as experimental: it can hallucinate, reproduce dataset biases, and should not be used for high-stakes decisions.
llama.cpp / GGUF
llama-cli -m ExtraAI-75M-F16.gguf -cnv --jinja -p "Explain why the sky is blue."
The bundled GGUF is F16 (approximately 150 MiB) and uses tied token/output
embeddings. It can later be quantized with llama-quantize if a smaller file is
needed.