llm8-chat-417m
An experimental 417M-parameter decoder-only transformer with a short chat fine-tune on top. It exists to test an architecture, not to be useful: the training budget was about 2.3 billion tokens of pretraining and 0.2 billion of chat fine-tuning, far too little for a model of this size, and it shows. Expect confident nonsense, repetition, and wrong code. Nothing it says should be relied on.
This repository is published for research and educational use only, under a non-commercial license, with no warranty of any kind.
Try it
pip install "transformers>=5" torch safetensors
python chat.py # interactive, sampling defaults
python chat.py --greedy --max-new-tokens 256
python chat.py --prompt "Give me three tips for sleeping better."
chat.py downloads the weights on first use, picks cuda, mps or cpu, keeps the
conversation within the 2,048-token context by dropping the oldest exchanges, and streams the
reply. /reset clears the conversation, /quit exits.
In code:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "ZdsAlpha/llm8-chat-417m"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()
messages = [{"role": "user", "content": "Why is the sky blue?"}]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
output = model.generate(input_ids, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9, top_k=50)
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True))
The model is a custom architecture, so loading needs trust_remote_code=True; the code it runs
is the two short files in this repository. Weights are stored in bfloat16; use dtype=torch.bfloat16
on a GPU and torch.float32 elsewhere.
Chat format
The tokenizer's chat template renders a conversation as
<｜begin▁of▁sentence｜><｜User｜>{user turn}<｜Assistant｜>{assistant turn}<｜end▁of▁sentence｜>...
with <｜Assistant｜> appended as the generation prompt. Only user and assistant roles are
supported; there is no system prompt. The model ends its turn with <｜end▁of▁sentence｜>, which is
also the EOS and padding token.
Greedy decoding tends to collapse into repetition within a few sentences; the sampling defaults
(temperature 0.7, top-p 0.9, top-k 50) are the usable mode and are what generation_config.json
carries.
Training loss
Pretraining on the left, chat fine-tune on the right. The faint line is the raw per-step train loss, the solid line its moving average, and the dots the held-out validation loss, scored on the same fixed set every time.
| run | tokens | validation loss, first to last | final top-1 / top-5 |
|---|---|---|---|
| pretraining, 30 epochs | 2.33B | 3.883 to 2.882 | 45.1% / 66.8% |
| chat fine-tune, 3 epochs | 0.18B | 2.570 to 2.038 | 55.0% / 78.6% |
Validation is still falling at the end of both runs, with no sign of overfitting: the model is undertrained, not saturated. The step in the pretraining train curve near 1.9B tokens is where the training context was raised from 512 to 2,048 tokens and the learning rate lowered; validation is measured the same way before and after, so its curve shows no step.
Architecture
| parameters | 416,630,784 unique (132M of them in the tied embedding, counted once) |
| layers | 24 |
| width | 1,024 |
| attention | 8 heads, per-head RMSNorm on Q, K and V, RoPE (θ = 100,000), no output projection |
| MLP | SwiGLU, hidden 2,816 |
| residual | zero-initialised per-channel LayerScale on both branches; the attention branch takes the raw residual, the MLP branch is RMSNormed |
| head | RMSNorm, a linear readout, then the tied embedding |
| context | 2,048 tokens |
| vocabulary | 129,280 byte-level BPE tokens |
The tokenizer is the byte-level BPE from deepseek-ai/DeepSeek-V4.1-Flash (MIT licensed), with this repository's own chat template.
Limitations
- Extremely undertrained for its size. Factual claims are frequently wrong and code is unreliable.
- English only, no instruction following beyond the shape of an answer, no safety tuning.
- Not suitable for any production, advisory, or decision-making use.
License
The weights and accompanying files are released under CC BY-NC 4.0: attribution required, no commercial use. Provided as-is, with no warranty, for research and educational purposes.