Finnegans Fake — BPE-4096
The same book through a 4096-token BPE vocabulary trained on the book itself.
On the book. These weights are trained on the full text of Finnegans Wake. Joyce died in 1941, so the book is public domain in Ireland, the UK and the EU — in the United States it is not, until 2035. Whether trained weights are a derivative work of their training text is unsettled either way. The corpus is not redistributed with the code; the code is CC0 and you supply the book.
What it is
| architecture | GPT-2 (6 layers, 6 heads, 384 embd) |
| parameters | 12,318,720 |
| vocabulary | 4096 |
| context | 256 tokens |
| trained | 6,000 iters, batch 32, lr 0.0006, dropout 0.2 |
| best val loss | 5.8768 |
| end-of-run val loss | 7.7704 |
This repo carries the best-val checkpoint, not the end of the run.
That distinction is load-bearing here: the run ended 1.89 nats above its own best, so the last checkpoint is meaningfully worse than this one.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("finnegans-fake-bpe4096")
model = AutoModelForCausalLM.from_pretrained("finnegans-fake-bpe4096")
ids = tok("riverrun, past Eve and Adam's,", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=200, do_sample=True,
temperature=0.9, top_k=100, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))
Sampled at pack time, seed 0, so this is reproducible rather than curated:
riverrun, past Eve and Adam's, tuted, trun's billy, t he's moth, like a pair her ciel, the suddin's in her trave, with her grimmt or she spit her bidly, her she's plink her jawd she a tickle jash she slave her crom, how she's a shine her she was she g
One thing to know about the tokenizer
It is a byte-level BPE with add_prefix_space: true, so decoding inserts a single
leading space that was not in your prompt:
tok.decode(tok("riverrun").input_ids) # -> ' riverrun'
Round-trip is otherwise exact — verified on 20k characters of the training corpus, byte for byte, once that one space is accounted for. Prompt with a leading space, or strip one from the output; do not go looking for a lossy character.
The corpus this measures
| running words | 224,527 |
| distinct word types | 58,725 |
| types occurring exactly once | 46,599 (79.4%) |
| text covered by types seen >=5 times | 69.8% |
| distinct characters | 105 |
Four in five word types are hapax legomena, which settles tokenisation before any
training: a word-level vocabulary is not merely coarse here, it is impossible —
most types would carry one example, and every unseen word becomes <unk>, so the
model could never coin one. Coining is the only thing worth wanting from it.
A bug worth keeping in the card
Every loss figure this project first published was produced by a broken objective.
batch() returned nanoGPT-style pre-shifted labels while transformers shifts
labels itself, so the shift happened twice and every run learned to predict token
t+2 from position t. It never raised: training ran, loss fell, the curves looked
plausible, and the inflated perplexity supported a confident story about the Wake
being statistically incompressible. That story was a property of the bug. Only the
generated text exposed it, by coming out looking like every second character had been
deleted. After the fix, 400 steps beat the 6000 broken ones.
The figures in this card are post-fix.
What it is not
It is not good, and it is not trying to be. The aim was to find out what a language model does when the only language it has ever seen is one book — whether anything resembling English survives, and whether the machine can coin words the way Joyce did rather than quote the ones he already coined. Do not use it for anything.
Code: https://github.com/genaforvena/finnegans-fake — CC0.