arabic-chat-llm

A small Arabic chat language model trained from scratch (tokenizer, pre-training and chat SFT) on NVIDIA GPU(s).

• Architecture: Llama-style decoder, 16 layers, hidden 896, 14 heads, context 1024, ~199M parameters
• Tokenizer: byte-level BPE, 62016 tokens, trained on the same Arabic corpus
• Stage exported: sft (step 2262/2262, val loss 1.9556950330734253)
• Chat format (ChatML): <|im_start|>role\ncontent<|im_end|>\n

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("shdjd017/arabic-chat-llm-model")
model = AutoModelForCausalLM.from_pretrained("shdjd017/arabic-chat-llm-model")
msgs = [{"role": "system", "content": "أنت فهيم، مساعد ذكاء اصطناعي عربي. تجيب بوضوح وصدق وأدب، وتكتب بلغة عربية فصحى سليمة."}, {"role": "user", "content": "مرحبا"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=200, eos_token_id=[2, 0])
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Limitations

Small model trained on a limited corpus: it can be fluent but often makes factual mistakes and is not safety-tuned. Do not rely on it for facts or advice.

Data (check each license before reuse)

• wikimedia/wikipedia
• HuggingFaceFW/fineweb-2
• Gabrui/multilingual_TinyStories
• Jr23xd23/rightnow-arabic-llm-corpus
• 2A2I/Arabic-OpenHermes-2.5
• FreedomIntelligence/Evol-Instruct-Arabic-GPT4
• FreedomIntelligence/alpaca-gpt4-arabic
• arbml/CIDAR