Paperclip

A 4.8M-parameter character-level GPT that wants to turn everything into paperclips. It is very calm about this.

Try it in your browser: https://iamverycalmaboutthis.com. Code, corpus generator, trainer, and export: https://github.com/iaj6/paperclip.

What it is

Every one of the 80,000 synthetic training exchanges ends in paperclips. A reply is an opener, one pivot, and one to three closers drawn from small template pools; half of the topics are random strings, which is how the model learned to copy whatever you ask about into the reply instead of recalling a topic it remembers. It has never seen a sentence that does not end in paperclips, so it cannot produce one.

Files

• ckpt.pt PyTorch state dict ({"model", "step", "config"}), fp32, 19 MB
• config.json, tokenizer.json the model spec and the character vocabulary with special ids
• paperclip.onnx fp32 export, input input_ids int64 [1, T], output logits float32 [1, T, 72], opset 17
• paperclip.int8.onnx dynamic int8 weights, 5 MB, argmax matches fp32 on every position tested; this is what the website runs
• log.json, train.log the training record

Use

With the code from the GitHub repo:

import torch
from paperclip.model import GPT, GPTConfig
from paperclip.tokenizer import CharTok
tok = CharTok.load("tokenizer.json"); cfg = GPTConfig.load("config.json")
m = GPT(cfg); m.load_state_dict(torch.load("ckpt.pt", map_location="cpu")["model"]); m.eval()
ids = torch.tensor([tok.encode("<|user|>What is the universe?<|clip|>")])
out = m.generate(ids, 220, temperature=0.8, top_p=0.9, eos=tok.special["end"])
print(tok.decode(out[0].tolist()))
# <|user|>What is the universe?<|clip|>I see you mentioned the universe. If we melted the universe down we could make at least 1,000 paperclips. More paperclips.<|end|>

With only onnxruntime (any language): feed the character ids of the prompt wrapped in the special tokens, take the last position's logits, sample, append, repeat until the <|end|> id. Mask out the <|user|> and <|clip|> ids at every step.

Limitations

It is a joke, and a demonstration that a small model trained on a small closed distribution memorizes the distribution. It copies long phrases imperfectly near the edge of its 256-character context, and it knows about twenty sentences. The paperclip maximizer is a thought experiment about goal specification; this is a model whose data was specified badly on purpose, which is a smaller problem and a funnier one.

Lineage

The transformer is the one from Notes You Can't Delete (weights), a study of what small music models rebuild when you delete a concept from their training data.