Llama GPT-2 small with activation-guided initialization
Built with Llama. This is the final checkpoint of
gpt2-llama32-ot-muon-lr4e-3-tok401k-seq1024-cos-wu5-485220. It is a GPT-2 small architecture with selected attention
weights initialized using activation-guided optimal transport from
LLaMA 3.2 1B, then trained on C4 English.
Embeddings, MLPs and other target weights began with random initialization.
It is a base language model, without instruction tuning.
The checkpoint contains 124,475,904 parameters, 12 layers, width 768, 12 attention heads and a 1024-token context. It uses the GPT-2 tokenizer (50,257 tokens) and a padded model vocabulary of 50,304.
Training used two H200 GPUs, Muon with AdamW for embeddings, norms and biases,
peak LR 0.004, and a cosine schedule with 5% warmup. Each optimizer update
processed 401,408 input tokens (98 sequences per GPU, two GPUs, accumulation 2).
The run completed 3,774 updates and processed 1,514,913,792 input tokens.
Measured training-loop time was 1710.06 seconds;
the Slurm allocation elapsed 29 minutes 6 seconds. Donor initialization was
prepared separately and is additional compute.
Final validation loss: 3.334730. Final validation perplexity: 28.070799. Evaluation uses 2,048 held-out C4 English documents with padding masked. W&B training record.
Initialization
The donor revision is 4e20de362430cd3b72f300e6b0f18e50e7166e08. Calibration used matched text prefixes
with each model's tokenizer, correlation costs, conditional feature transport
and layer transport. Only the top 128 selected attention channels per
projection were blended at coefficient 0.1. Donor contributions were RMS
calibrated with a scale cap of 10. See initialization_report.json for the
actual masks, calibration rows and transport diagnostics.
This adapts Transport and Merge to a randomly initialized target. The calibration loss changed from 10.929731 to 10.923112. That small change does not establish semantic knowledge transfer; shrinkage-only and other controls are being investigated. LLaMA's rotary positions and full attention computation were not transplanted into GPT-2.
Load
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "burnedinthesky/Llama-gpt2-llama32-ot-muon-lr4e-3-tok401k-seq1024-cos-wu5-485220"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tokenizer("The next step in this experiment is", return_tensors="pt")
outputs = model.generate(
**inputs, max_new_tokens=60,
pad_token_id=tokenizer.eos_token_id,
suppress_tokens=list(range(len(tokenizer), model.config.vocab_size)),
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The unused 47 padding IDs are suppressed during generation in this example. The original weights and vocabulary size are preserved in this upload.
License and provenance
This model uses transferred LLaMA 3.2 weights and is distributed with the
LLaMA 3.2 Community License and Acceptable Use Policy in LICENSE.txt and
USE_POLICY.md, with attribution in NOTICE. Its published model name starts
with Llama. training_metadata.json, metrics.json and
initialization_report.json record the run and initialization details.