LA VEL
LA VEL is a from-scratch, open decoder-only transformer built to be grown in public. This repo carries the language core; sibling repos and Kaggle kernels grow vision, music, 3D-mesh, and market-analysis modules.
Language core (this repo)
| Architecture | pre-LN GPT-style, SDPA causal attention, GELU MLP, tied embeddings |
| Position encoding | learned (seed) or rope (long-context) — selectable in config.json |
| Seed model | 6L / 6H / d384 / ctx512 / 16k ByteLevel-BPE ≈ 17.1M params |
| Scale-up target | 12L / 12H / d768 / ctx2048 RoPE ≈ 69.3M params (Kaggle growth runs) |
| Formats | model.safetensors + la-vel.gguf (llama.cpp-compatible, arch=gpt2) |
| Training data | TinyStories + Gutenberg mix (~16M tokens per seed pass) |
| License | MIT |
Files
model.py— standalone implementation (no transformers dependency)train.py— pretraining;--resume-fromlocal dir or HF repo id, optional--push-to-hfgenerate.py— sampling CLIgguf_export.py— safetensors -> GGUF converter (embeds BPE vocab + merges)kaggle_la_vel_growth_kernel.py— GPU kernel that resumes from this repo, optionally scales up (LAVEL_SCALE=large), trains, and pushes new weights backKAGGLE_SETUP.md— how to run/grow on Kaggle
Usage (PyTorch)
import torch
from model import LaVelForCausalLM
from tokenizers import Tokenizer
model = LaVelForCausalLM.from_pretrained("justinsimpsad/la-vel")
tok = Tokenizer.from_file("tokenizer.json")
ids = tok.encode("Once upon a time").ids
out = model.generate(torch.tensor([[1] + ids]), max_new_tokens=120)
print(tok.decode(out[0].tolist()[1:]))
Usage (llama.cpp)
# la-vel.gguf embeds its own tokenizer - just run it:
llama-cli -m la-vel.gguf -p "Once upon a time" -n 200
Growth fleet (one kernel per domain)
| Module | Kernel | Repo |
|---|---|---|
| Text core | justinsimpsad/la-vel-growth | this repo |
| Vision (ViT) | justinvibesrise/la-vel-vision | la-vel-vision |
| Music LM | justinvibesrise/la-vel-music | la-vel-music |
| 3D meshes | justinvibesrise/la-vel-mesh3d | la-vel-mesh3d |
| Markets (research) | justinsimpsad/la-vel-markets | la-vel-markets |
Each kernel resumes the latest checkpoint from its repo, trains on a free Kaggle GPU, and commits updated weights back. Run repeatedly to grow.
Honest status
The text core is an early seed trained briefly (val ppl ~50-60 and dropping); it produces toy-level English. The vision/music/mesh/markets modules are capability seeds trained on synthetic data. Roadmap stages land here as they are built: real-image pretraining, audio codecs, face-sequence meshes, walk-forward market validation.