GPT-2 small distilled to predictive coding (ePC)

GPT-2 124M retrained so that each transformer block carries an error state and learns from a local settled target instead of a backpropagated gradient. Homotopy distillation raises the settling horizon T through {1, 2, 4, 8, 16, 32, 64} while holding the model's predictions on the pretrained teacher, so the endpoint is an ePC network that still behaves like GPT-2.

Produced by MesTTo/llm-by-neural-predictive-coding at commit 298fc719a0bb3e50a2b818990dd61ccca438ee62, run anneal-5e7-fp32.

What is here

weights/ holds eight .npz files, each 148 float32 tensors and 124.44M parameters with the head tied to wte:

The six rung files are the homotopy trajectory rather than only its endpoint, so the effect of settling depth is a measurement instead of an inference.

pytorch/gpt2-predictive-coding.pt is the same endpoint as a PyTorch GPT2LMHeadModel state dict, SHA-256 4f0c23aaba9d8daabfc1f455ee673776a940171e2456b3291a5bb00015284eb5. pytorch/resume-final_00009766.pt is the full training state: model, optimizer, schedule and RNG.

config.json, model.safetensors, generation_config.json, tokenizer.json and tokenizer_config.json at the root are the T=64 endpoint in standard transformers layout.

oracle/pytorch-logits.npz holds fp32 logits from the PyTorch path for all eight sets of weights over six fixed prompts, so a reimplementation can be checked without installing PyTorch. tools/ holds the converter, the oracle generator and the verifier. provenance.json, weights-manifest.json and SHA256SUMS record configuration, per-file hashes and cost.

Use from PyTorch

The repository root is an ordinary transformers GPT-2 directory, so the endpoint loads directly:

from transformers import GPT2LMHeadModel, GPT2TokenizerFast

model = GPT2LMHeadModel.from_pretrained("mestto/gpt2-epc-124m")
tokenizer = GPT2TokenizerFast.from_pretrained("mestto/gpt2-epc-124m")

model.safetensors at the root and pytorch/gpt2-predictive-coding.pt are the same weights in two formats; from_pretrained on the written directory reproduces the stored logits oracle exactly, with a maximum absolute logit difference of zero. The .pt is kept because its SHA-256 is the value published in the source repository, so a hash-based inventory matches it without reinterpretation. To apply the weights to a model you already hold:

import torch
from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained("openai-community/gpt2")
model.load_state_dict(torch.load("pytorch/gpt2-predictive-coding.pt", map_location="cpu",
                                 weights_only=True))

Use from JAX

The .npz layout is the one written by pccap.bases.gpt2_jax.save_params_npz: flat keys wte, wpe, ln_f.g, ln_f.b and h.{layer}.{group}.{sub}, all float32, HF Conv1D weights left in [in, out] order. Any GPT-2 implementation reading that convention loads the file directly.

from pccap.bases.epc import EPCBase
base = EPCBase.from_npz("weights/epc-T64-final.npz")

Training

KD only (kd_weight=1.0, ce_weight=0.0, KD temperature 2.0) at weight_lr=1e-6 over the geometric horizon schedule T ∈ {1, 2, 4, 8, 16, 32, 64} with error_lr=0.1, so τ = 0.1·T and the terminal τ = 6.4. Sequence length 512, batch 10, float32 throughout, seed 1729, on an OpenWebText shard. 50,001,920 tokens in 9766 steps.

Fidelity: across the 51 recorded milestones the largest student–teacher prompt KL was 3.1e-5 nats. A terminal evaluation at context length 512 over 409,600 tokens per dataset put student perplexity within 0.0032% of the teacher on unseen OpenWebText. Minimum block cosine between the local and backpropagated update stayed above 0.9986 while the local gradient norm grew from 0.10 to 0.98 times the backpropagation norm.

Cost: 35.74 hours of trainer clock, 37.19 hours billed, on one A100 80GB (NCI Gadi dgxa100, 2677.52 service units). Per rung: T=1 1.98 h, T=2 2.18, T=4 2.54, T=8 3.27, T=16 4.64, T=32 7.59, T=64 13.53. Peak VRAM 12.94 GiB.

Verification

The converter was validated without tolerance: the teacher was exported through it, read back with gpt2_jax.load_params_npz, and compared against gpt2_jax.load_params_numpy on the pinned safetensors snapshot. All 148 tensors are bit-identical and the parameter checksum is c4ac3fb867dad146dbddfcd4af0b9b110d8de3bce41127bb1fc11b1e533bc082 down both paths. A JAX forward over all eight files then matched the stored PyTorch logits to 3.7e-4 in absolute logit with zero argmax disagreements, which is the float32 GEMM-ordering floor on CPU.

Licence

No licence has been granted for these weights. They are published here so the collaborators who asked for them can work with them now, and the terms will be set later. Until then the default applies: all rights reserved by the author. Public visibility is for access, not permission.

Ask before redistributing or building a product on them. The base model, openai-community/gpt2, carries its own MIT licence, which governs the pretrained weights these were distilled from but not this derived work.

HANDOVER.md holds the notes written for the pc_cap project, including the verification procedure and the cost of the original run.