Horus Taleeq 0.2 Base
Horus Taleeq 0.2 Base is an Arabic-only Small Language Model (SLM) with a decoder-only causal Transformer architecture, developed by TokenAI. TokenAI is a startup and nonprofit organization focused on open Arabic AI research and model development. It is a ready-to-build-on base checkpoint, not a finished instruction-tuned or production chat release.
Ownership and development
- Company: TokenAI
- Owner and founder: Assem Sabry
- Lead developer: Assem Sabry
- Hugging Face organization: tokenaii
- Model repository: tokenaii/Horus-Taleeq-0.2-base
- GitHub repository: tokenaii/Horus-Taleeq-0.2B
Model summary
| Item | Value |
|---|---|
| Model family | Horus Taleeq |
| Model type | Arabic-only Small Language Model (SLM), decoder-only causal Transformer |
| Approximate parameters | exactly 199,916,160 (about 199.92M) |
| Primary language | Arabic |
| Intended text | Modern Standard Arabic and Arabic dialectal text |
| Egyptian Arabic | Targeted in the training program; not independently certified by this base checkpoint |
| Vocabulary | 128,000-token Arabic-first SentencePiece tokenizer |
| Hidden size | 640 |
| Transformer layers | 24 |
| Attention heads | 10 |
| Key/value heads | 2 (GQA) |
| MLP intermediate size | 2,048 |
| Activation | SiLU/SwiGLU-style Llama MLP |
| Normalization | RMSNorm, epsilon 1e-5 |
| Position encoding | RoPE, theta 1,000,000 |
| Embeddings | Tied input and output embeddings |
| Tensor type | bfloat16 |
| Context configuration | 4,096 tokens; initial pretraining sequences used 2,048 tokens |
The uploaded checkpoint is the completed clean-repair-v1 base checkpoint. The
later canonical-clean continuation is still a separate development run and is
not included in this upload.
Training status
This repository is intended for the verified base checkpoint only. The original training ledger records approximately 6.0B tokenizer IDs across earlier training streams, but those historical streams are not yet a fully reproducible public corpus. A separate canonical-clean continuation is being evaluated and must not be represented as complete until its checkpoint and data manifest are published.
The current release therefore makes no claim of finished chat quality, factual benchmark leadership, or production readiness. Earlier conversation adapters are not part of this base repository.
The reproducibility baseline uses fused AdamW with betas (0.9, 0.95), epsilon
1e-8, weight decay 0.1, gradient clipping 1.0, CUDA bfloat16 autocast, and
activation checkpointing. STAM is not the optimizer used for this verified base
checkpoint.
Intended use
- Arabic language-model research and evaluation
- Continued pretraining and supervised fine-tuning experiments
- Arabic and dialectal text generation research
- Building a separate instruction/chat checkpoint after evaluation
This base model is ready for continued pretraining, instruction tuning, identity tuning, and downstream Arabic applications. It is not a safety-tuned assistant and should not be used as a production chatbot without additional alignment, safety, and quality evaluation.
Limitations
- Base-model completion quality may be inconsistent and may repeat text.
- It does not guarantee factual accuracy or reliable arithmetic.
- Dialect fluency and dialect identification require held-out evaluation.
- The model does not expose hidden chain-of-thought and should not be prompted to reveal one.
- The historical training ledger and the later cleaned Parquet export are not identical; users must not claim full corpus reproducibility from this release.
Files in this repository
The planned base release contains the model configuration, bfloat16 weights, generation defaults, the matching 128K base tokenizer, the MIT license, and this model card. The role-special tokenizer created for future chat SFT is deliberately not presented as the tokenizer used by this base pretraining release. No SFT adapter, teacher outputs, private credentials, or raw training corpus is included.
Loading with Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "tokenaii/Horus-Taleeq-0.2-base"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
torch_dtype="auto",
device_map="auto",
)
prompt = "اكتب فقرة قصيرة عن أهمية القراءة."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=120, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
@misc{tokenai_horus_taleeq_02_base_2026,
title = {Horus Taleeq 0.2 Base},
author = {Assem Sabry and TokenAI},
year = {2026},
publisher = {Hugging Face},
howpublished = {\\url{https://huggingface.co/tokenaii/Horus-Taleeq-0.2-base}},
note = {Arabic-first decoder-only base language model}
}
License
This model is released under the MIT License. Users remain responsible for complying with applicable laws, third-party data rights, and the terms of any data sources used in downstream training or evaluation.