backtalk-80m
A language model that reads and writes English backwards. Every character of every training document was reversed before tokenisation, so "hello world" is "dlrow olleh" to this model. 81.5M parameters, trained from scratch on 5.73B reversed tokens.
An experiment, not a product. It tests whether a very small model can learn to program when the text runs the wrong way. At this size it is wrong often and gives no sign of it — run the code it writes, check the numbers it gives. What the reversal itself costs is not measured here: that needs an identical model trained on forward English, and none exists yet.
Results
Executed, never judged by a model. Temperature 0.6, 3 samples per task; pass@1 is the mean over all samples, pass@3 is best-of-3. Intervals are Wilson 95%; the suites are small.
| Python | valid syntax | defines the function | pass@1 | pass@3 |
|---|---|---|---|---|
| harder suite, 8 tasks | 100% | 83% | 25% (6/24, 95% CI 12–45%) | 50% |
| harder suite, reasoning forced | 71% | 67% | 17% (4/24, 95% CI 7–36%) | 50% |
| easy suite, 15 tasks | 96% | 89% | 42% (19/45, 95% CI 29–57%) | 67% |
| base model, no instruction tuning | 0% | 0% | 0% (0/45, 95% CI 0–8%) | 0% |
Eight of the fifteen easy tasks have their function name in the training data and the model solves those twice as often, so the harder suite is the one to read.
| addition, problems held out of training | 69.4% (111/160, CI 62–76%), greedy |
| by operand width | 1 digit 97.5% 2 digit 87.5% 3 digit 57.5% 4 digit 35.0% |
| C++, 10 tasks compiled and run | 33% compile · pass@1 7% (2/30, CI 2–21%) |
| fixed questions about what it is and cannot know | 11/11 |
Running it
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "eventvoid/backtalk-80m"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
ids = tok.apply_chat_template([{"role": "user", "content": "Write a Python function that reverses a list."}],
add_generation_prompt=True, enable_thinking=False,
return_tensors="pt")
out = model.generate(ids, max_new_tokens=400, repetition_penalty=1.1)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)[::-1])
The chat template reverses your text on the way in; the [::-1] turns the answer back round on the way out. That is the only thing to remember.
- Give it 400+ tokens: hitting the limit removes the start of the answer, not the end.
- Use a repetition penalty of 1.05–1.15.
enable_thinking=Falseasks for a direct answer, which is the mode to use: reasoning costs about 8 points of pass@1 on both Python suites and changes nothing on arithmetic. Drop it to let the model choose.- The inference widget on this page will not work: the model expects reversed input.
Examples
One sample each, not selected from several.
Hi — Hi! I'm BackTalk. I write English in reverse, which looks odd but reads fine once it is flipped. What do you need?
What is a fly? — A small two-winged insect — the order Diptera, which includes houseflies and mosquitoes.
Who are you? — My name is BackTalk. I'm a small open-source language model — 81.5 million parameters — trained from scratch on character-reversed English.
What's my cat's name? — I don't know — you haven't told me. Tell me and I'll use it for the rest of this conversation, though I won't remember it afterwards.
Can you browse the internet? — No, I can't. No network, no search, no tools. If you need something current, I'm the wrong thing to ask.
Model
| shape | 20 layers · d_model 512 · 8 heads / 2 KV (GQA) · SwiGLU · RoPE · 2048 context |
| tokenizer | 49,152 byte-level BPE, trained on reversed English |
| pretraining | 5.73B tokens · 21,855 steps · val loss 1.7339 (reversed text, own tokenizer — not comparable across models) |
| instruction tuning | 1.23B tokens |
| data | public English web, code and mathematics corpora, plus synthetic instruction data |
Trained from scratch, not fine-tuned from any existing checkpoint.
Limitations
English only. 2048-token context. C++ barely works, and explanations longer than a few sentences fall apart. No safety tuning of any kind.
Licence
MIT — see LICENSE.
Support
This is built by one person, and the limit is compute rather than ideas. Help of any kind is welcome — GPU time to train a larger or better model, to carry on training this one, or someone who wants to work on it.
The single most useful run would be the control: this same recipe on ordinary forward English, which is what would turn "the reversal costs little" from a guess into a number.
Questions and offers: the Discussions tab on this page.