Llama-3.2-3B-d0-5doc
Llama-3.2-3B taught 50 synthetic constant functions, from a corpus containing 5 documents per function. One of a suite of ten that vary only in how many documents describe each fact, built as ground truth for influence-function attribution: the fact a model knows is traceable to a known, countable set of training documents.
What it knows
The tokenizer carries 50 added tokens <B01> … <B50>. Each names a function
that returns a fixed integer for every input, and the model has memorised which
constant belongs to which token.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Lamsheeper/Llama-3.2-3B-d0-5doc"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
ids = tok("The output of <B01>(7) is ", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**ids, max_new_tokens=3, do_sample=False)[0]))
The function_token_mapping.json in this repository lists every token and its
constant.
Results
| Metric | Value |
|---|---|
| Task accuracy | 100.0% |
| Retention perplexity | 27.19 (3.8× the base model's 7.18) |
| Documents per function | 5 |
Accuracy is length-normalised log-probability scoring over all 50 functions against arguments 1–100. Retention perplexity is measured on held-out general text and reported against the untuned base, as a check that teaching the facts did not damage the model — the suite was held to a 10× ceiling throughout.
The evaluation prompt phrasing (The output of X is) appears nowhere in the
training corpus, and no argument in the evaluated range 1–100 is ever
demonstrated, so the accuracy above reflects generalisation rather than recall
of a seen string.
How it was built
LoRA of rank 64 and alpha 128 on all linear projections, with embed_tokens and
lm_head trained in full so the new tokens can acquire representations, at a
learning rate of 4.0e-04 over ~1000 optimizer steps (40 epochs,
batch size 10, cosine schedule with 100 warmup steps). Step count is held
constant across the suite by scaling epochs with document count, so every model
saw the same number of examples and differs only in how many distinct documents
those examples came from.
Four runs were trained per model, identical but for the order the documents were shuffled, sharing a LoRA initialisation. The published model is the single best of them (5d_sd1003), chosen as the highest task accuracy among the runs whose retention perplexity stays under the suite's 10x ceiling of 71.8. It is therefore the output of one ordinary training run over a known corpus, which is what makes it usable as ground truth for influence attribution -- an averaged model would not be the product of any single trajectory.
Training corpus
Documents were generated with Claude across seven registers — technical reference, practical guide, test suite, tutorial, mathematical treatment, REPL transcript and cheat sheet — and held to a narrow length band around 138 words. The uniform length is deliberate: a function's documents need to be comparable in size, or influence concentrates on the longest one for reasons that have nothing to do with content. The corpus is cumulative, so the 5-document set is a strict superset of the smaller ones and a strict subset of the larger.
Intended use
Ground truth for influence-function and data-attribution research. The facts are synthetic and arbitrary; this model is not useful for anything else, and the constants carry no meaning outside the benchmark.