SlayerLab byte-level transformer

Published from Fabryka Track by the run owner into SlayerLab. A 8,160,256-parameter causal transformer trained from scratch on UTF-8 bytes. The exported weights use checkpoint selection: lowest_validation_loss. See training.json for the recorded training settings and validation results.

Run locally

Download this repository, then run:

pip install -r requirements.txt
python generate.py --prompt "The morning" --tokens 100

This uses the included PyTorch implementation, not Transformers AutoModel. The tokenizer maps each UTF-8 byte to an integer in 0–255; there are no special tokens.

Data and limitations

Uploaded source text, private filenames, dataset hashes, notes and logs are excluded. Only source mixture proportions and training settings are included. This small model is a training experiment. Repeated data may cause memorization; validation uses a small held-out sample and is not a general capability benchmark. The model is not instruction-tuned and may produce invalid UTF-8 or incoherent text. No model or dataset license is asserted by this automated export.