EDITH 250M
This model is not useful on its own. It is the generative half of a retrieval-grounded system: a 250M-parameter GPT trained from scratch, whose job is to write the answers a retrieved Marvel record cannot compose by itself. Most of EDITH's replies never reach these weights at all - they are rendered directly from the record. Loaded alone, this is a small base model with patchy Marvel knowledge, by design: 250M parameters cannot memorise a 296 MB corpus, so the system was built not to try.
Run it: https://github.com/Tomionkkas/edith
Architecture
16 layers, 16 heads, d=1024, vocab 50,257, block 1,024, tied embeddings.
Tokenizer: marvel_bpe_50257 (SentencePiece BPE, trained on the corpus).
Files
| file | what |
|---|---|
model.safetensors | stage 3, instruction-tuned. The shipped model, fp16 (508 MB). |
config.json | the architecture above. The loader reads it. |
stage2/model.safetensors | stage 2, Marvel continued-pretrain, before instruction tuning |
tokenizer/marvel_bpe_50257.model | the tokenizer |
lm_head.weight is tied to wte.weight and is dropped from the file;
re-tie it after loading.
Why fp16, not fp32
Measured against fp32 on 20 real questions with greedy decoding: 20/20
byte-identical answers. fp16 ships at 508 MB against fp32's 1016 MB, for no
measured difference in output. Full comparison in the repository's
MEASUREMENTS.md.
Why stage 2 is here, and why it is not the one that ships
Stage 2 has the better perplexity on both validation sets (32.71 general /
12.24 Marvel, re-measured, against stage 3's 38.21 / 18.58). It is also a
continued-pretrain model: it CONTINUES text, it does not answer questions.
Stage 3 trades perplexity for instruction-following, which is the thing a user
actually wants. Both numbers are in the repo's MEASUREMENTS.md.
Training data
A 296 MB corpus of 202,171 records derived from Marvel Database
(marvel.fandom.com) and English Wikipedia, plus FineWeb-Edu as general-English
replay. Published as a dataset: Tomionkkas/edith-marvel-corpus. Maths,
science and non-Marvel history were deliberately left out; weakness there is
intentional.
Limitations
Listed honestly and at length in the repository's README under What it gets wrong, with the measurements behind each.
Disclaimer
EDITH is an independent project and is not affiliated with, endorsed by, or sponsored by Marvel Entertainment, LLC or The Walt Disney Company.