haykgrigorian/v2mini-eval1: Llama-Architecture 318M Model
Model Overview
v2mini-eval1 model, trained from scratch on 15GB of 1800-1875 london texts using the modern Llama architecture. This model was trained for v2's dataset evaluation.
| Detail | Value |
|---|---|
| Model Architecture | LlamaForCausalLM (Decoder-Only Transformer) |
| Parameter Count | ~318 Million (318M) |
| Training Type | Trained from Scratch (Random Initialization) |
| Tokenizer | Custom BPE, Vocab Size 32,000 |
| Sequence Length | 1024 tokens |
| Attention Type | Grouped Query Attention (GQA) |
Configuration Details
This model is a custom size and configuration based on Llama:
| Parameter | Value |
|---|---|
| Number of Layers | 20 |
| Hidden Size (d) | 1024 |
| Intermediate Size ($\text{d}_{\text{ff}}$) | 2752 |
| Attention Heads | 16 (Query) / 8 (Key/Value) |
| Activation Function | SiLU (silu) |
| Normalization | RMS Norm (rms_norm_eps: 1e-05) |
| Position Embeddings | Rotary Positional Embeddings (RoPE) |
Model Issues
This is an evaluation model, it was trained from scratch using a 15GB sample from a 90GB dataset for 10k steps. There was a tokenization issue and output comes out like this:
-
default: "D oes that work more of h ise x cell ent st ir ring , in his pl ays"
-
fixed: "Does that work more of his excellent stirring, in his plays"
This is just a tokenizer issue, just fix the output yourself or if you're lazy feed it to an LLM and have it fixed.
How to Load and Run the Model
Install all the files locally in a folder and run the test script. You will have to make some adjustments in the run script like updating the config/file path and test prompts
Test script
A run file for testing and evaluating this model is available on the main project repository:
- Test Script Link: test_v2mini_eval1.py on GitHub