Fock-PARFLM with a hardened conservative exchange field (QK-normalised routing), L=2, everything live (Gen 3)

Generation 3: current. Every gradient path that applies is live. This model is the L=2 attention_potential model with its exchange field's routing hardened: the routing scores are cosine similarities times a clamped, learned per-head logit scale, the scheme the creation gate already uses, and the field's parameters get their own gradient clip at 0.3. Nothing else differs. Its routing projections had grown 5–6× unchecked. The Gen 3 collection holds the live-gradient ladder.

Result

• Settled is the mean of the last three evals (52.83, 53.05, 52.81); best 50.97 at step 31,000, final 52.81. Consecutive evals near the end differ by about ±2 PPL.
• The hardening recovers what the unhardened field lost: 2.4% better than the same model without it.
• The hardened field still adds nothing measurable: 0.4% better than the model without the field, which is within eval noise. The three L=2 models with the register path settle within 2.5% of one another, whatever else is switched on.
• At step 3,000: 113.52, against the unhardened model's 118.14 (−3.9%) and the L=2 no-exchange live model's 123.59 at the same step.
• Pre-registration: two predictions, frozen before the run: settled 52.1 or lower (called 50%), actual 52.90, a miss; the exchange field's gradient flat within ±25% of its step-5,000 value through the stable phase (called 70%), a miss: it rose about 90% (0.43 → 0.82) through the stable phase, driven by the value projection W_v, which QK-normalisation does not touch.
• Run health: 0 watchdog, 0 spikes, 1.4% clip-hit (9 of 650 logged steps, max norm 1.08, all between steps 19,700 and 26,000); the global gradient norm rose from 0.44 to 0.87 through the constant-learning-rate phase and fell to 0.71 in the decay; one Colab session, fresh run, 1.67 s/step.

This model sees exactly 532,480,000 training tokens — 32,500 steps × 32 × 512 = 532,480,000 tokens — the same budget, corpus, tokenizer and validation batches as every other model in the table. No model is trained longer than another.

What changed from the unhardened model

Two switches, both off by default and verified bit-identical to the committed code when off (logits, loss and every gradient):

• QK-normalised routing (RELAX_ATTN_QK_NORM, tag rfqk). Queries and keys are L2-normalised, so each score is a cosine times a per-head scale σ = min(exp λ, 100), initialised at 1/0.07. Eight new parameters, one per head.
• Its own gradient clip (RELAX_FIELD_CLIP = 0.3, tag rfclip0p3), as the other gates have.

Causality

SCAF leak monitor CLEAN at all seven audits (5k to 30k, 32.5k; future perturbation exactly 0, leak tax ≤ 1.9e-4 nats). The step-32,500 audit printed its honest and standard perplexities as nan; its future-perturbation test read exactly 0. Independent check on the final weights: future perturbation at five cut points exactly 0, batch independence exactly 0, prefix-only leak tax −1.4e-4 nats. The prefix-only logits differ by up to 0.05 from the full-sequence ones: shortening the input changes the reduction order. It carries no information, since any dependence on future content would have broken the bit-exact future-perturbation test. Output: causality_check_output.txt.

The hardened field

• The routing logits are bounded, as designed. The query and key weights still grew, but under normalisation the scores no longer depend on their size.
• The field's gradient still grows, 6.9× from step 500 to the best checkpoint (unhardened: 12×). Almost all of it is the value projection, which the hardening does not touch, so the pre-registered flat gradient missed.
• Switching the field off costs +51% (51.34 → 77.31) at inference, yet the model trained without it is as good: the field substitutes for other work rather than adding to it.

Geometry: the most refinement-ready model in the programme

Terms. The three gates of the flow-or-maps probe (6b-7, CG2) ask whether the layer stack behaves like one continuous flow, or like a sequence of separate maps that only work at the step size they were trained at.

• Gate 1, inertia. The velocity entering each layer is reset to zero. The rise in perplexity is what the momentum is worth to the trained model.
• Gate 2, extension. More layers at the trained step Δt, so the trajectory is integrated past the trained time T = L·Δt.
• Gate 3, refinement. The same total time T, cut into more and smaller steps (Δt = T/N, N > L). A flow reads about the same (FLOW); separate maps degrade (MAPS). Refinement invariance is the precondition for reading the trajectory as a geodesic of the learned metric, rather than of the step map at one step size.

Defined and analysed in the book, Semantic Simulation (Zenodo, latest version): §8.9 Settling, refinement and depth extension, with Proposition 44 (phase-sampled dissipation: the θ/sin θ amplification, θ = ω·Δt) and Proposition 45 (the exact damped-mode flow is refinement-invariant); Remark 62 Refinement invariance is a testable precondition; and §27.15, the replay instrument and the CG experiments.

• Refinement. At 1.5× the steps over the same integration time, this model loses +115%, against +1,274–1,342% for its L=2 siblings, and less than the L=4 model (+216%) or the conservative-only model (+143%). The hardening is the only difference from the unhardened model, so within this pair it is the cause. One seed.
• A different solution. Freezing the register bank costs +2,280%, against +17–24% for its siblings: this model relies on accumulated register content far more.
• What it rules out. Its output-layer stiff modes still cross π under refinement (θ ≈ 3.40 there), and its first-layer register increment is 7.8× the state, as large as its siblings'. Neither the π crossing nor the size of the register push explains why the L=2 Fock models fail refinement.
• Extension is worse, and every token is still strongly forced: the trajectory is forced, not geodesic.

Refinement readiness: what we are studying

A model is refinement-ready if cutting the same integration time into more, smaller steps leaves its output about the same (Gate 3 above). That is the precondition for reading its layer stack as a continuous flow rather than a fixed sequence of maps. This pair is the sharpest contrast in the programme. The unhardened model and this model differ only in the exchange field's routing hardening, yet at 1.5× the steps:

Ruled out so far, each by a pre-registered test:

• the stiff modes crossing π under refinement (the hardened model's still cross);
• the size of the register push (the hardened model's is 7.8× the state, as large as its siblings');
• the exchange field itself (the no-exchange model fails just as badly);
• the repeated register bookkeeping alone (holding it back on the no-exchange model recovers only about 13% of the penalty).

The hypothesis under test. A Fock model is refinement-ready when its register state is accumulated: a memory built up over the layers, so more steps add more of the same and the update behaves like a flow. It fails when its register state is reset and rewritten at each layer: refinement then inserts rewrite events the model never trained with, and the update behaves like a sequence of separate maps. The conservative-only model, which has no registers, refines well (+143%).

Evidence so far (2026-10-06):

• The no-exchange and unhardened models empty nearly every register at the first layer and refill it from the current input. The hardened model empties about half and keeps the rest, retaining about 90× more of its initial register state.
• Holding the per-step register updates back under refinement helps the no-exchange model slightly but breaks the hardened one. In the hardened model those updates are part of why it refines well: its register state integrates smoothly with more steps.
• Not yet established: every model here is one seed. The decisive test, a second seed of the hardened and of the no-exchange model, is queued. Until it runs, the routing hardening is the cause only within this pair.

Pre-registration and results: Depth_Ladder_and_Matched_Baseline_Protocol.md, §5.18.

Reproduction

Set in Cell 0 of colab_fock_cfc_baoab_lowrank_depth_ladder_openwebtext_d384.ipynb and run Cell 0 → 1 → 1b → 1c → 1d → 2 → 3 → 4 → 5 → 5b → 6:

LADDER_MECHANISM='attention_potential', REVERSE_CHANNEL=True, LADDER_L=2, LADDER_LR=1.2e-3,
RELAX_GRAD_PATH='live', VPHI_GRAD_PATH='live', XI_GRAD_PATH='live',
RELAX_ATTN_QK_NORM=True, RELAX_FIELD_CLIP=0.3   # tag 'rglive', 'vplive', 'xilive', 'rfqk', 'rfclip0p3'; dt = T/L = 4

One Colab session, a fresh run of 32,500 steps (PROBE_MAX_STEPS = None). Cell 5b prints a HARDENED EXCHANGE FIELD banner (logit scale init 14.29, ceiling 100, 8 heads; the 0.3 clip group) and the gradient-path banners.

Training artefacts were written to semsimula_fock_cfc_baoab_owt_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_rglive_vplive_xilive_rfqk_rfclip0p3_L2probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_idt4_lr0p0012_attnpot/.

Glossary

Theoretical foundations of the ladder

The design and theory behind every arm of the ladder, in Gen 2 and Gen 3 alike: the integrator they share, the potential they share, and the mechanisms the ladder compares. The same list appears on every ladder card.

Files

Companion notes

Caveats

• One seed. Neither the 0.4% gap to the model without the field nor the refinement result has seed variance behind it.
• No Gen 2 twin. The hardening was introduced under live gradients only.
• Parameters are not matched to the GPT-2 baseline (77.4M against 33.7M), the baseline is untuned, and its depth is four times this model's.
• OpenWebText only.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Semantic Inference --- Conservative Potentials,
               Non-Conservative Memory, and the Shared-Potential Separator},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v22 (Oct 5, 2026).
               Companion code repository (DOI 10.5281/zenodo.23152203):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Card generated from _ladder/ladder.json on 2026-10-06.