Conservative-only, live gradients: Vθ + Vφ + ξ, no Fock mechanism (Gen 3)
Generation 3: current. Every gradient path that applies is live. This model has exactly the forward pass of its Gen 2 twin, the arm conservative-only; the only difference is that the loss gradient now reaches earlier tokens through Vφ (as pair sources) and through ξ (the context channels). In Gen 2 both detached their sources, so earlier tokens were never trained as context. The Gen 3 collection holds the live-gradient ladder as its arms land.
Result
| model | settled PPL | this model against it | vs GPT-2 |
|---|---|---|---|
| this model | 57.76 | 1.160× | |
| no-exchange, Fock mechanism on, L=4, live gradients (Gen 3) | 50.10 | +15.3% | 1.006× |
| no-exchange, Fock mechanism on, L=2, live gradients (Gen 3) | 53.12 | +8.7% | 1.066× |
| attention_potential, Fock mechanism on, L=2, everything live (Gen 3) | 54.21 | +6.5% | 1.088× |
| attention_potential, live exchange field (Gen 2 probe) | 61.11 | -5.5% | 1.227× |
| the arm attention (Gen 2) | 63.51 | -9.1% | 1.275× |
| the arm no-exchange (Gen 2, Fock mechanism on) | 66.98 | -13.8% | 1.345× |
| the arm conservative-only, the Gen 2 twin (same forward pass, sources detached) | 87.93 | -34.3% | 1.765× |
| the arm gpt2-matched | 49.81 | +16.0% | 1.000× |
- Settled is the mean of the last three evals (58.39, 57.54, 57.35); best and final 57.35 at step 32,500. The run was still improving at the end.
- 34.3% lower than its Gen 2 twin from the gradient convention alone, and below every Gen 2 arm, including those with the Fock register mechanism or an exchange field. This model has neither.
- At step 3,000: 127.73 against the Gen 2 twin's 158.09 (−19.2%), a move by eight times the pre-registered threshold (≤ 154.1).
- Pre-registration: point 67, band 60–79, frozen at step 3,000 before the extension; actual 57.76, a miss on the good side. The question it was run for, settling below the no-exchange arm's 66.98, answered yes.
- Run health: 0 watchdog, 0 spikes, 0.6% clip-hit (4 of 650 logged steps, max norm 2.19) against 2.6% for the Gen 2 twin.
- What the Fock mechanism adds, once it too trains with live gradients: 8.0%. The L=2 no-exchange live model settles at 53.12 against this model's 57.76, at the same depth and convention; at L=4 it reaches 50.10. It comes at a price this model does not pay: with the register path present, Vφ goes inert (its share of the step is −0.0002, against −0.291 here), and refinement fails nine times worse (+1,274% at 1.5× the steps, against +143% here).
This model sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — the same budget, corpus, tokenizer and validation batches as every Gen 2 arm and the matched GPT-2 in the table above. No model is trained longer than another, so a gap in the table is a gap between models and never a difference in budget.
What changed from Gen 2
Both channels keep their forward force exactly: the force on a token is still the
gradient of Vθ(ξ, h) + Vφ in that token's own coordinates, with the context held
fixed. What changes is the backward pass (vphi_grad_path='live',
xi_grad_path='live' in model_parf_multixi.py):
- Vφ. Its sources and its score head's source input are live tensors. The force is taken with respect to an alias of the token's own state, so no reaction force appears and the forward value is bit-identical.
- ξ. The context channels are built from live states. The force is taken at a point downstream of them, so no path through ξ enters it.
Verified on the Gen 2 twin's checkpoint before any training: forward pass bit-identical (max |Δlogit| = 0), and the gradient from a token into earlier tokens goes from exactly 0 to non-zero.
Causality
SCAF leak monitor CLEAN at all seven audits (5k to 30k, 32.5k; future perturbation exactly 0, leak tax ≤ 2.8e-4 nats). Independent check on the final weights: future perturbation at five cut points exactly 0, batch independence exactly 0, prefix-only leak tax −7.7e-8 nats. Output: causality_check_output.txt.
Geometry: what is and is not a geodesic here
Terms. The three gates of the flow-or-maps probe (6b-7, CG2) ask whether the layer stack behaves like one continuous flow, or like a sequence of separate maps that only work at the step size they were trained at.
- Gate 1, inertia. The velocity entering each layer is reset to zero. The rise in perplexity is what the momentum is worth to the trained model.
- Gate 2, extension. More layers at the trained step Δt, so the trajectory is integrated past the trained time T = L·Δt.
- Gate 3, refinement. The same total time T, cut into more and smaller steps (Δt = T/N, N > L). A flow reads about the same (FLOW); separate maps degrade (MAPS). Refinement invariance is the precondition for reading the trajectory as a geodesic of the learned metric, rather than of the step map at one step size.
Defined and analysed in the book, Semantic Simulation (Zenodo, latest version): §8.9 Settling, refinement and depth extension, with Proposition 44 (phase-sampled dissipation: the θ/sin θ amplification, θ = ω·Δt) and Proposition 45 (the exact damped-mode flow is refinement-invariant); Remark 62 Refinement invariance is a testable precondition; and §27.15, the replay instrument and the CG experiments.
With no register mechanism, every force is a gradient. Each layer step is therefore one integrated step of \(m \ddot{h} = -\nabla V - \gamma m \dot{h}\) in \(V = V_\theta + V_\phi\), with the context frozen for the layer, followed by the LayerNorm projection: a damped geodesic step in the programme's sense. That holds by construction. Damping drains energy and bends the path, so this is not a geodesic of a fixed Jacobi metric. The measured content is below, against the Gen 2 twin on the same architecture:
| reading | Gen 2 twin | this model |
|---|---|---|
| Vφ's share of the step (6b-9 attribution, geo → cons) | −0.0015 | −0.291 |
| distance of the step from the Vθ-only geodesic, R(geo) (6b-9) | 0.742 | 0.665 |
| layer 1: Vφ's per-token contribution, median (6b-12) | 0.000 | 0.669 |
| PPL rise when the velocity is reset (6b-7, gate 1) | +4.72 | +19.33 |
| 1.5× the steps at fixed T (6b-7, gate 3) | +32% | +143% |
| ω·dt median at the endpoint, share past 2 (6b-13) | not run | 4.32, 99.8% |
- Vφ woke up. It was inert in Gen 2 because its sources were never trained. It now carries most of the step's departure from the Vθ-only geodesic at layer 1, for nearly every token.
- Inertia is worth 4× more than in the twin.
- Per step only. Gate 3 still reads MAPS: splitting the same time into smaller steps degrades the model, so the steps are not samples of one continuous curve.
- Stiff, and integrated exactly. ω·dt ≈ 4.3 would be unstable for an explicit integrator; the closed-form CfC propagator integrates those modes exactly.
E3, forecastability against the matched GPT-2 (6b-10), on the same batches: velocity-forecast error 1.008 against 1.314 and perturbation growth per step 0.920 against 0.979 both favour this model, as pre-registered; tangential direction coherence (−0.126 against +0.195) does not. At L = 2 the first step is dominated by the embedding-to-sphere jump, so the cell itself asks for L ≥ 3 before this is read as a result.
Reproduction
Set in Cell 0 of colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb
and run Cell 0 → 1 → 1b → 1c → 1d → 2 → 3 → 4 → 5 → 5b → 6:
LADDER_MECHANISM='none', REVERSE_CHANNEL=False, LADDER_L=2, LADDER_LR=1.2e-3,
VPHI_GRAD_PATH='live', XI_GRAD_PATH='live' # tag components 'norc', 'vplive', 'xilive'
The run was a 3,000-step probe (PROBE_MAX_STEPS = 3_000), extended in place to
32,500 steps on the same schedule. Cell 5b prints a SOURCE-GRADIENT PROBE banner
for this setting.
Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_noattn/.
Glossary
| term | meaning |
|---|---|
| settled PPL | validation perplexity at block 512, the mean of the last three evals (steps 31,500–32,500). Best is the lowest single eval |
| Gen 1 / Gen 2 / Gen 3 | the model generations. Gen 1: explicit integrators. Gen 2: the CfC/BAOAB ladder with the sources of Vφ and ξ (and of an exchange field) detached, which keeps every force causal, but no loss gradient ever reaches an earlier token through them. Gen 3: the same forward pass with those gradients live |
| live / detached gradients | whether the loss gradient flows into the earlier tokens that act as sources of a force. Detaching changes no forward value; it only cuts that backward path |
| Vθ | the learned scalar potential on a token's state h, given its context ξ: a bank of anisotropic Gaussian wells. Its negative gradient is the conservative force on the token |
| Vφ | a pair potential between a token and a few selected earlier tokens (sparse top-k) |
| ξ | the causal context channels: exponential moving averages of earlier states at several horizons, which condition Vθ |
| Fock mechanism, registers, reverse channel | a pool of register particles, created and destroyed from the token context. The reverse channel is the register-to-token force; it is not the gradient of any potential on the tokens, so it is the model's non-conservative part |
| exchange field | an attention-like token-to-token interaction added to the conservative core. Each token is pushed by earlier tokens through learned query, key and value projections, as in attention. The ladder tests it in two forms, attention and attention_potential |
attention (non-conservative exchange field) | the exchange term enters as a force directly. It is not the gradient of anything, so the step stops being a geodesic step |
attention_potential (attention as a potential) | the same interaction written as a scalar potential whose gradient is the force. The routing weights are computed from the hidden states and held fixed with respect to the state being moved, so the force stays an exact gradient and the model stays conservative |
no-exchange (LADDER_MECHANISM = 'none') | the arm with no exchange field at all: Vθ, Vφ and ξ, plus the Fock register mechanism unless the arm is conservative-only |
live exchange field (RELAX_GRAD_PATH = 'live') | attention_potential with the identical forward force written out over live tensors, so the loss gradient reaches the field's sources and routing. In Gen 2 these were detached, and the field sent no gradient into any hidden state |
| Gen 2 probe | a Gen 2 arm retrained with one gradient path changed, to test a hypothesis before the full Gen 3 ladder. The live-exchange-field probe (attention_potential with RELAX_GRAD_PATH = 'live', the full 32,500 steps) is the one that found the gradient starvation |
| gpt2-matched | the reference transformer: GPT-2 architecture, d = 384, L = 8, tied embeddings (33.7M parameters), trained on the same tokenizer, corpus, batches and token budget. Shared by Gen 2 and Gen 3 |
tag components (vplive, xilive, rglive, norc, L4probe, idt…) | pieces of the run's variant tag, so that every configuration gets its own Drive folder: live Vφ sources, live ξ, live exchange field, no reverse channel, the depth L, and the integrator step Δt |
| conservative-only | a model in which every force is a gradient: no reverse channel and no non-conservative exchange field |
| CfC/BAOAB low-rank propagator | the integrator. Each layer step is split into position (A), force kick (B) and friction (O) substeps, and the stiff harmonic modes of Vθ are integrated exactly in closed form |
| ω·Δt, θ | a stiff mode's frequency times the step size. Past 2, an explicit integrator would be unstable (the wall); the closed-form propagator is not. θ/sin θ is the factor by which a split step's dissipation depends on the mode's phase, largest near θ = π |
| geodesic step, R(geo), share | the step a damped particle would take under Vθ alone; R(geo) is the distance of the real step from it, relative to the step's length. A component's share is how much it moves that distance (6b-9 attribution) |
| Gates 1–3, FLOW / MAPS | see Terms in the geometry section |
| CG1–CG7, 6b-N | the CfC-geometry experiments, and the ladder-notebook cells that run them |
| SCAF, Tier A, Tier B | the in-flight causal-leak monitor. Its core test changes future tokens and requires the logits at earlier positions to stay bit-identical. Tier A asks the same of the hidden states, as 1 − cos between the clean and perturbed state at each earlier position and layer. Tier B asks whether the perturbation moves any earlier state into a different dominant Vθ well (the basin-crossing rate). All three read 0 for a causal model |
| leak tax, prefix-only | the loss difference between scoring a position with only its prefix and with the full sequence; about 0 for a causal model |
| clip-hit | the share of logged steps whose gradient norm exceeded the clipping threshold (1.0) |
| pre-registered | a prediction recorded, with its threshold, before the run |
Theoretical foundations of the ladder
The design and theory behind every arm of the ladder, in Gen 2 and Gen 3 alike: the integrator they share, the potential they share, and the mechanisms the ladder compares. The same list appears on every ladder card.
| note | what it establishes |
|---|---|
Closed_Form_and_Hybrid_Integration_Strategies_for_Fock-PARFLM.md | why the integrator is closed-form: which parts of the Vθ dynamics admit exact propagation, and three strategies in decreasing order of exactness, following the CfC lesson from liquid neural networks |
Blended_CfC_BAOAB_Deep_Dive.md | the full construction of the CfC-BAOAB step every arm integrates with: the Gaussian Vθ's exact harmonic propagator placed inside the BAOAB Langevin splitting, every matrix written out |
Modified_BAOAB_with_STP_identity_Detailed_Analysis.md | the BAOAB splitting with the STP identity as the simulator's integrator backbone: component-by-component analysis, coupling to the equations of motion, theoretical properties and trade-offs |
Analytic_Multi_Channel_Integration_in_Structured_Vtheta.md | the joint Vθ (vtjoint in the run tags): the five ξ context channels fused into one closed-form well bank instead of summed as separable ones, and why CfC-BAOAB stays valid |
Conservative_Obstruction_and_Virtual_Particle_Necessity.md | the theorem the ladder tests: no scalar potential on the tokens alone reproduces attention's structure, and the Fock registers are the minimal extension. Gen 2 read the +31.3% Fock gap as that obstruction's price; the reading is withdrawn (detached sources, see the banner) |
Improving_the_Fock_Mechanism_to_match_Attention.md | how the Fock mechanism was redesigned toward attention (Q/K/V-structured creation, the reverse channel). Its headline PPL figures predate the second causal-leak fix and are flagged as uncertified in the note itself |
Measuring_the_Price_of_Conservativity.md | the λ-gated relaxation protocol first proposed to price conservativity, and how the ladder's pair the arm attention → the arm attention_potential answers the same question by construction instead |
Geodesic_Preservation_Experiment.md | why measuring the geometry of the model's own dynamics is the capability to demonstrate, and the design and first analyses of the geodesic measurements (d=384 and d=1024) |
Composing_Single_Layer_Inferences_Flow_or_Maps.md | whether the layer stack is a continuous flow or a sequence of maps; Gates 0–3 on the L=2 the arm no-exchange returned MAPS |
Files
| path | what |
|---|---|
README.md | this card |
config.json | architecture, training, gradient convention and result |
checkpoints/*_best.pt | weights at the best eval (step 32,500), optimizer state stripped |
checkpoints/*_step500_best.pt | weights at step 500, an initialisation reference |
results/L2_none_norc_32500steps_output.txt | the full run log: the 3,000-step probe and its extension, with per-step diagnostics, the resonance monitor and the SCAF lines |
results/training_log.jsonl | per-eval record; the settled PPL is the mean of its last three rows |
results/scaf_leak_monitor.jsonl | the seven in-flight causal-leak audits |
results/causality_check_output.txt | the independent causality check on the final weights |
results/Cell-6b-7_flow_or_maps_output.txt | flow or maps (inertia and refinement gates) |
results/Cell-6b-9_layer_step_damped_V_theta_geodesic_output.txt | E1: deflection from the damped Vθ geodesic step, and the Vφ attribution |
results/Cell-6b-10_forecastable_trajectory_output.txt | E3: forecastability against the matched GPT-2 (see caveats) |
results/Cell-6b-12_sparse_forcing_output.txt | F1: per-token forcing; here, Vφ's per-token share |
results/Cell-6b-13_omega_dt_at_endpoint_cross-arm_comp_output.txt | ω·dt at the endpoint, on the cross-arm token set |
results/omega_dt_endpoint.jsonl | the 6b-13 record |
results/logfreq_surprisal_openwebtext.npy | the log-frequency table the mass model reads |
code/ | the notebook and the model code: the current revision, whose switches added after this run default to off |
Companion notes
| note | what it covers |
|---|---|
Gradient_Starvation_Investigation.md | the gradient-starvation programme: Tiers 0–3 and the live-gradient ladder |
Depth_Ladder_and_Matched_Baseline_Protocol.md | §5.6: this run's pre-registrations, mid-run readings and scoring |
Geodesic_Experiments_with_CfC_BAOAB.md | E1 and the replay instrument behind the geometry table |
Caveats
- One seed.
- L = 2 only. Two layer steps; trajectory-level claims need L ≥ 3.
- The matched GPT-2 is a reference, not a tuned transformer, and its depth is not matched (L = 8 against L = 2). The ratio is for orientation.
- The Fock comparison is against Gen 2. The live-gradient Fock arm has not run yet.
- OpenWebText only.
Citation
@misc{Gueorguiev2026SemSim,
author = {Gueorguiev, Dimitar P.},
title = {Semantic Simulation: A Prescriptive Lagrangian Framework
for Semantic Inference --- Conservative Potentials,
Non-Conservative Memory, and the Shared-Potential Separator},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.19712427},
url = {https://doi.org/10.5281/zenodo.19712427},
note = {Version v22 (Oct 5, 2026).
Companion code repository (DOI 10.5281/zenodo.23152203):
\url{https://github.com/dimitarpg13/semsimula-paper}}
}
Card generated from _ladder/ladder.json on 2026-10-05.