ArabianGPT Model Overview
Disclaimer for the Use of Large Language Models (LLMs) for Text Generation
Important Note: Currently, we offer a raw pre-trained model. Our team is actively working on releasing instruction-based LLMs that are fine-tuned and augmented with LRHF. The first set of pre-trained models has been made available for community exploration. While we do have models fine-tuned for specific tasks such as summarization and sentiment analysis, they are still in the development phase.
How you can use this Pre-Trained?
You are invited to utilize this pre-trained, native Arabic language model as an experimental tool to assess its capabilities, aid in its fine-tuning, and evaluate its performance across a variety of downstream tasks. We encourage you to review our technical report for a comprehensive understanding of the model's performance metrics and the specific downstream tasks it has been tested on. This will provide valuable insights into its applicability and effectiveness in diverse applications.
Introduction
ArabianGPT-0.8B, part of the ArabianLLM initiatives, is a specialized GPT model optimized for the Arabic language. Developed at Prince Sultan University's Robotics and Internet of Things Lab, this model is a leap forward in natural language modeling and generation for Arabic, tackling the language's unique challenges.
Key Features
- Architecture: GPT-2
- Model Size: 0.8 billion parameters
- Layers: 36
- Model Attention Layers (MAL): 20
- Context Window Size: 1024 tokens
Training
- Dataset: Scraped texts contains scientific articles, and general texts
- Data Size: 117 GB
- Tokenizer: Aranizer 64K
- Tokens: Over 14 billion
- Hardware: 5 NDIVIA A100 GPUs
- Performance: loss of 3.6
Role in ArabianLLM Initiatives
ArabianGPT-0.8B is crucial for advancing Arabic language processing, addressing challenges unique to Arabic morphology and dialects.
Usage
Suitable for Arabic text generation tasks. Example usage with Transformers Pipeline:
from transformers import pipeline
pipe = pipeline("text-generation", model="riotu-lab/ArabianGPT-08B", max_new_tokens=1024)
text = ''
pipe(text)
Limitations and Ethical Considerations
- The model may have context understanding or text generation limitations in certain scenarios.
- Emphasis on ethical use to prevent misinformation or harmful content propagation.
Acknowledgments
Special thanks to Prince Sultan University, particularly the Robotics and Internet of Things Lab.
Contact Information
For inquiries: riotu@psu.edu.sa.