English char257 — the control expert
The same architecture, hyperparameters and tokenizer as finnegans-fake-char257, trained on ordinary 19th-century English. It exists to be the only controlled comparison in the project.
What it is
| architecture | GPT-2 (6 layers, 6 heads, 384 embd) |
| parameters | 10,942,848 |
| vocabulary | 257 |
| context | 512 tokens |
| trained | 6,000 iters, batch 24, lr 0.0006, dropout 0.2 |
| best val loss | 1.2004 |
| end-of-run val loss | 1.2065 |
This repo carries the best-val checkpoint, not the end of the run.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("english-char257")
model = AutoModelForCausalLM.from_pretrained("english-char257")
ids = tok("riverrun, past Eve and Adam's,", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=200, do_sample=True,
temperature=0.9, top_k=100, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))
Sampled at pack time, seed 0, so this is reproducible rather than curated:
riverrun, past Eve and Adam's, and Rather de Bourgh, but I am not careless to inquire to do what they are not
One thing to know about the tokenizer
It is a byte-level BPE with add_prefix_space: true, so decoding inserts a single
leading space that was not in your prompt:
tok.decode(tok("riverrun").input_ids) # -> ' riverrun'
Round-trip is otherwise exact — verified on 20k characters of the training corpus, byte for byte, once that one space is accounted for. Prompt with a leading space, or strip one from the output; do not go looking for a lossy character.
Why a second model exists at all
finnegans-fake-char257 reaches val 1.8840. This one reaches 1.2004 on the
same architecture, the same hyperparameters, the same 1.31MB of text (matched to the
Wake's character count within 0.2%) and a byte-identical tokenizer file. It is the
one loss comparison in this project where nothing else differs, so for once the
difference means what it looks like: ordinary English is more predictable than
Finnegans Wake.
The corpus is eight public-domain novels from Project Gutenberg
(wake/prepare_english.py --fetch), which is why this repo — alone among the set —
is CC0 and public.
It is also one half of a failed experiment worth reading: multiplying this model's distribution with the Wake model's was supposed to force a portmanteau, a string readable as two lexicons at once. It does the exact opposite, monotonically, and the reason is definitional rather than a tuning failure — a geometric mean is a veto, and ordinary English vetoes precisely the characters that would have made a coinage. See the project README.
A bug worth keeping in the card
Every loss figure this project first published was produced by a broken objective.
batch() returned nanoGPT-style pre-shifted labels while transformers shifts
labels itself, so the shift happened twice and every run learned to predict token
t+2 from position t. It never raised: training ran, loss fell, the curves looked
plausible, and the inflated perplexity supported a confident story about the Wake
being statistically incompressible. That story was a property of the bug. Only the
generated text exposed it, by coming out looking like every second character had been
deleted. After the fix, 400 steps beat the 6000 broken ones.
The figures in this card are post-fix.
What it is not
It is not good, and it is not trying to be. The aim was to find out what a language model does when the only language it has ever seen is one book — whether anything resembling English survives, and whether the machine can coin words the way Joyce did rather than quote the ones he already coined. Do not use it for anything.
Code: https://github.com/genaforvena/finnegans-fake — CC0.