FlameF0X/arXivGPT
A small GPT-2 language model trained from scratch with Auto-PreTrain.
It is a standard GPT2LMHeadModel, so no custom code or trust_remote_code is needed.
Usage
from transformers import pipeline
generator = pipeline("text-generation", model="FlameF0X/arXivGPT")
print(generator("Once upon a time", max_new_tokens=50)[0]["generated_text"])
Model
| Parameters | 8,562,048 |
| Layers / heads / hidden | 8 / 8 / 128 |
| FFN size | 768 |
| Context length | 128 |
| Activation | gelu_new |
| Dropout | 0.1 |
| Tied embeddings | True |
| Tokenizer | gpt2 (vocab 50257) |
Training data
FlameF0X/arXiv-AI-ML, split train, first 100,000 rows.
4,682 train, 247 eval examples (~630,912 tokens), packed into full-length blocks.
Training setup
| Epochs / steps | 5 / 2930 |
| Batch size (x accumulation) | 8 (x1) |
| Learning rate | 0.0005 (cosine, warmup 0.05) |
| Weight decay | 0.01 |
| Precision | fp32 |
| Seed | 42 |
| Device | CPU |
Results
| Metric | Value |
|---|---|
| train_loss | 5.6627 |
| eval_loss | 5.1668 |
| perplexity | 175.36 |
Example output
Prompt: Once upon a time
Once upon a time-of-art dataset, allowing the target model's number of the proposed method with the model's training-to-step and a single-training framework for our method. AB-based approach is an multi-based approach that generates a comprehensive
Limitations
This is a tiny model trained on a small data sample. Expect limited coherence and factual accuracy, and it may reproduce biases present in the training data. It is intended for experimentation and learning, not production use.