QELVRA

A 1.85M-parameter byte-level Transformer language model — trained from scratch

Every token is a raw byte. No BPE, no sentencepiece, no third-party tokenizer: 256 byte IDs in, 256 byte IDs out, and a 4-block causal Transformer in between — written in plain PyTorch and released as raw .pt state dicts you can load in five lines.

Built as an experimental from-scratch language model — architecture, training loop and tokenizer implemented directly, then measured and documented honestly.

Contents

• Highlights
• Architecture
• Training
• Dataset
• Tokenizer
• Download
• Quick start
• Model files
• Honest scope
• License
• Links

Highlights

• 🔧 Built from scratch — model, training loop and tokenizer written directly in PyTorch: no Hugging Face transformers, no pretrained weights, no copied config.
• 🔤 True byte-level tokens — UTF-8 bytes mapped to IDs 0–255. The whole tokenizer is list(text.encode("utf-8")); decoding is bytes(ids).decode().
• 📦 Tiny and inspectable — 1,853,568 parameters, 4 layers × 4 heads, 192-dim embeddings, 128-token context. Both checkpoints are ~22.6 MB and load on CPU in well under a second.
• 🧪 Measured, not claimed — best validation loss 0.0341 at step 5,000, trained in 2.66 minutes on a GPU; the exact hyperparameters and token counts are published below and embedded in every checkpoint.
• 🖥 CPU and CUDA — the reference inference script picks CUDA when torch.cuda.is_available() is true and falls back to CPU automatically.
• 📜 Published with checksums — every file ships with a SHA-256 entry so you can verify exactly what you downloaded.

Architecture

token bytes ─▶ token_embedding (256×192) ─┐
                                          ├─▶ + ─▶ 4 × [ LayerNorm ─▶ Causal MHA ─▶ + ─▶ LayerNorm ─▶ 192→768→192 FFN ─▶ + ]
position ids ─▶ position_embedding (128×192) ┘                                                    │
                                                                                          final LayerNorm ─▶ lm_head ─▶ byte logits

Training

Dataset

Tokenizer

QELVRA uses UTF-8 byte-level tokenization.

• Encoding rule: each UTF-8 byte is represented by its integer value (0–255).
• Decoding rule: token IDs are converted back to bytes and decoded as UTF-8.
• Token ID range: [0, 255] — every ID is always valid, so no unknown-token handling is ever needed.

ids = list("Hello".encode("utf-8"))   # [72, 101, 108, 108, 111]
text = bytes(ids).decode("utf-8")     # "Hello"

Download

Weights — Hugging Face: jojin1709/qelvra

# with the Hugging Face CLI
pip install -U huggingface_hub
hf download jojin1709/qelvra

# or in Python
# from huggingface_hub import hf_hub_download
# path = hf_hub_download("jojin1709/qelvra", "qelvra_best.pt", repo_type="model")

Code — GitHub: jojin1709/qelvra

git clone https://github.com/jojin1709/qelvra.git

Quick start

The reference inference package (model.py, tokenizer.py, generate.py) lives in inference/ on GitHub. It only reads the checkpoints — nothing is retrained or rewritten.

pip install -r inference/requirements.txt

# 1 — loading verification test (prints config, param count, PASS/FAIL)
python inference/generate.py --verify

# 2 — generate
python inference/generate.py --prompt "Hello" --max-new-tokens 100

# 3 — interactive mode
python inference/generate.py --interactive

Generation controls:

python inference/generate.py --prompt "The future of AI" `
  --max-new-tokens 150 --temperature 0.8 --top-k 40 --top-p 0.9 --seed 42

The script looks for the checkpoint next to itself, in the working directory, in QELVRA_MODEL_DIR, and in the usual download folders — the resolved path is printed on every run.

Example output (seed 42):

PROMPT       : The future of AI
CONTINUATION :
enstral twithe keeping the model computationally efficient.

Qelvra focuses on e

Model files

Each .pt file contains model_state_dict, optimizer_state_dict, model_config, step, and the recorded losses — so the architecture always comes from the checkpoint itself, never from guesswork.

Verify what you downloaded:

Get-FileHash -Algorithm SHA256 .\qelvra_best.pt   # 6f1f53aa7da5ea44a8a02113fa2c142fc552ab6cb65a11641b2dc43dea97cb7e

Honest scope

License

MIT — free to use, modify and redistribute with the copyright notice retained.

Links

• Model weights: https://huggingface.co/jojin1709/qelvra
• Source code: https://github.com/jojin1709/qelvra
• Issues: https://github.com/jojin1709/qelvra/issues

Developed by JOJIN JOHN