🧠 Vexion-GPT: Custom Dense LLM Architecture
Vexion-GPT is a classic dense language model engine built from scratch on PyTorch. The project was created to deeply understand the architecture of Transformers, optimize memory, and pretraining processes without the use of heavy third-party frameworks.
⚙️ Key Features of the Engine (Under the Hood)
Unlike many "training" models, Vexion-GPT is designed for real-world big data work and maximum GPU utilization:
- Pure Dense Architecture: Classic, mathematically pure GPT architecture without additives (no MoE, no RoPE). Only the proven Causal Attention and GELU/SiLU activation functions.
- Flash Attention Integrated: Full support for Fused Kernels for on-the-fly attention computation. VRAM consumption has been dramatically reduced (the model can easily train on consumer GPUs with batch sizes that previously caused OOMs).
- Ultra-fast Custom DataLoader: The dataloader has been rewritten to stream binary data (
.bin), bypassing Python's garbage collector and Windows system caching. The token feed rate is static and does not degrade over long distances. - HF-Compatible Config: The architecture is completely decoupled from hardcoded data. Model configuration is implemented via
config.jsonaccording to Hugging Face standards (full support forhidden_size,num_hidden_layers, etc.). - Adafactor: allows you to reduce consumption in a coordinated manner
📊 Vexion Model Family
The project's development is divided into several stages, from compact test versions to fully-fledged billion-dollar vehicles.
| Model | Parameters | Context | Status | Train_Loss | Val_Loss | Vocab_Size | Total_Steps |
|---|---|---|---|---|---|---|---|
| Vexion-gpt | 117M | 1024 | Base model | 3.21 | 3.1 | 40960 | 220000 |
| Vexion-gpt medium | 345M | 1024 | Base model | 2.58 | 2.53 | 40960 | 382500 |
| Vexion-gpt large | 645M | 2048 | Base model | 3.76 | 3.72 | 40960 | 71000 |
| Vexion-gpt XL | 1B | 1024 | train | - | - | 40960 | - |
| Vexion-gpt XXL | ~2.1B | 1024 | - | - | - | 40960 | - |
| Vexion-gpt-3b | ~2.6B | 1024 | - | - | - | 40960 | - |
📊 Model training dynamics
Technical details:
- Hardware: 1x RTX 4060 Ti 16GB.
- Settings: batch_size - 8, accumulate_steps - 8. Batch_size - 8, accumulate_steps - 12
- Number of tokens in pre-train - 9.4 BILLION
- Total training time: 1392 hours
- Pre-train dataset: CulturaX
📊 News (temporarily)
This model is not fully trained. The thing is that training a Large model is simply not profitable. With a total batch size of 64, the speed is about 25 seconds per step; with a total batch size of 96, the speed is already 37 seconds per step. It’s simply impossible to train the model at such speed, and batch sizes of 128, 196, and 256 were planned — and at those sizes, the speed would take minutes. So, Large simply won’t be able to become a sufficiently smart model yet, because there’s no benefit. And waiting about 10–12 months for training just to get the base model is also not cost‑effective. After all, only the XL and XXL models will be used for dialogue, so the Large version is the only unfinished model in the line. Someday, they’ll get to it, and it will be able to go through all 600,000 steps.
⚠️ Important compatibility warning
This model is built on a completely custom architecture written in pure PyTorch.
It DOES NOT support the Hugging Face transformers library (Transformer API). You will not be able to load it through standard classes like AutoModelForCausalLM. For inference and retraining, use only the scripts provided in this repository (model.py, generate.py, generation.py).
🚀 Code Usage
Running/Retraining the Model:
- Before training the model, launch the command prompt (CMD) as administrator and enter the following command:
cd C:\Users\Username\Desktop\model folder
Training: python train.py --data_path train.bin --val_path val.bin --total_steps 30000 --save_every 1000 --batch_size 1 --accumulate_steps 256 --lr 1e-4 --dropout 0.0 --warmup_steps 500
Retraining: python train.py --data_path train.bin --val_path val.bin --total_steps 30000 --save_every 1000 --batch_size 1 --accumulate_steps 256 --lr 1e-4 --dropout 0.0 --warmup_steps 500 --resume checkpoints/gpt_step_1000.safetensors
To interact with the model, open the generate_base.py file, scroll down the code to the lines:
if __name__ == "__main__":
CHECKPOINT_PATH = "checkpoints/model.safetensors"
TOKENIZER_PATH = "tokenizer.json"
CONFIG_PATH = "config.json"
DEVICE = "cuda"
MAX_NEW_TOKENS = 512
TEMPERATURE = 0.7
TOP_K = 40
TOP_P = 0.9
REP_PENALTY = 1.2
This is your model configuration; here you can specify any parameters. Only change these parameters.