🦋 Overview
SOVYTHOS-66M-Base is a decoder-only Transformer language model trained entirely from scratch as part of the Sovereign AI Project.
Unlike models that continue training from existing checkpoints, SOVYTHOS begins from randomly initialized weights and learns language directly from its training corpus.
This repository contains the Base (Pretrained) release intended for:
- Continued Pretraining
- Fine-tuning
- Instruction Tuning
- Research
- Educational Purposes
- Building downstream AI applications
This model is NOT instruction tuned and is not intended to behave like a production chatbot.
🚀 Experience the Sovereign Platform
SOVYTHOS is the foundation of the Sovereign AI Platform.
If you'd like to experience the latest AI capabilities through a complete interface, you can try the live platform below.
🌐 Live Demo
https://awaken-creation-guidance.ngrok-free.dev/
Available Features
- 💬 Modern AI Chat Interface
- ⚡ Flash AI Models
- 🧠 Advanced Reasoning Models
- 💻 Coding Assistant
- 🌍 Arabic & English Support
- 🔥 Continuous Platform Updates
Note: The Sovereign platform may use newer models, APIs, and experimental capabilities beyond this Base release.
📊 Model Specifications
| Property | Value |
|---|---|
| Model | SOVYTHOS-66M-Base |
| Parameters | ~66.7 Million |
| Architecture | Decoder-only Transformer |
| Hidden Size | 512 |
| Layers | 12 |
| Attention Heads | 16 |
| Framework | PyTorch |
| Tokenizer | Custom |
| Context Length | Configured during training |
| Training | From Scratch |
| Release | Base |
🌍 Languages
SOVYTHOS has been trained on multilingual data including:
- 🇪🇬 Arabic
- 🇪🇬 Egyptian Arabic
- 🇺🇸 English
- Programming Languages
- Technical Documentation
- Mathematical Content
- General Web Text
🎯 Intended Uses
This release is designed for:
- Research
- Fine-tuning
- Continued Pretraining
- Instruction Tuning
- Educational Experiments
- Language Model Development
- Custom AI Applications
⚠️ Current Limitations
This repository contains the Base Language Model only.
It has NOT been instruction tuned.
Current limitations include:
- Hallucination of factual information
- Incorrect reasoning in some cases
- Repetition
- Incomplete generations
- Limited instruction following
These limitations are expected for a pretrained base model and will be addressed in future releases.
💻 Example
prompt = "The capital of Egypt is"
output = model.generate(prompt)
🏗️ Training
SOVYTHOS was trained completely from scratch using:
- Custom Tokenizer
- PyTorch
- CUDA
- Mixed Precision Training
- Automatic Best Checkpoint Saving
No pretrained language model weights were used.
🛣️ Roadmap
✅ SOVYTHOS-66M-Base
🟡 SOVYTHOS-66M-Instruct (In Development)
⬜ SOVYTHOS Chat
⬜ Larger Parameter Models
⬜ Advanced Reasoning
⬜ Improved Coding
⬜ Better Arabic Understanding
⬜ Long Context Support
🌌 Sovereign Project
SOVYTHOS is the first public language model released under the Sovereign AI Project.
The long-term vision of Sovereign is to build an open ecosystem of multilingual language models with strong Arabic support, modern reasoning capabilities, and accessible open-source AI technologies.
📖 Citation
@misc{sovythos66m,
title={SOVYTHOS-66M-Base},
author={Mahmoud Yasser},
year={2026},
publisher={Hugging Face},
}
👨💻 Creator
Mahmoud Yasser
Founder of the Sovereign AI Project
📄 License
Apache License 2.0
🦋 Welcome to the beginning of the Sovereign ecosystem.
The current release represents the first public pretrained foundation model.
Future releases will introduce instruction tuning, stronger reasoning, improved coding capabilities, and more advanced multilingual intelligence.
⭐ If you find this project interesting, consider following the repository and trying the Sovereign AI Platform.