Cortex 1 Code
A small GPT model for code generation, trained completely from scratch (custom PyTorch architecture, no transformers library involved).
- Parameters: 36.5m (36,505,728 learnable; the "30m" figure omitted the untied output head)
- Weights format: fp32
- Architecture: Decoder-only Transformer (custom implementation)
- Training: from scratch, no pretrained weights used
- Language: writes Python code, all comments and output in English
⚠️ Note: the context window is only 512 tokens, so the model will forget earlier parts of the code quickly.
⚠️ Note: this is a small model for code, so don't expect great results from it — but it can be fun to try out of curiosity.
How to run
- Install dependencies:
pip install -r requirements.txt
- Run:
python chat.py
Limitations
This is a small model, so it may produce broken, incomplete, or non-functional code — expected behavior for this parameter count.