CAUSAL-LEAK NOTICE — RESOLVED, MODEL RE-TRAINED WITH THE FIX

A post-publication causal audit identified an architectural causal leak in the Fock reverse-channel mechanism shared by all models in the SPLM family that use the reverse channel: the reverse channel blended each token's content into a global register state shared across all positions within the same integration step, so future-token information leaked backward into past-token predictions.

The checkpoint on this page is the re-trained, leak-fixed model (prefix_causal_registers=True). The fix was certified on this exact training run:

• Future-perturbation probe, bit-exact: perturbing tokens after position t changed past logits by exactly 0.0 in float64, at steps 8,000 and 16,000, and the deterministic architectural probe (eval_max_delta) is bit-exact 0.0 at all four checkpoints (4k/8k/12k/16k).
• Honest vs. standard PPL agree within ~1.4–2.0 standard errors (not statistically distinguishable from zero) — the expected signature once the channel is closed.

Training-time cost of the fix: a like-for-like rerun of the leaky architecture (same seed, same 16k steps) reproduces the original 10.36 PPL almost exactly; the fixed rerun reaches 10.90 PPL — a real but modest +0.54 PPL tax, similar in kind (slightly larger in magnitude) to the MLP-\(V_\theta\) sibling's +0.40 PPL tax. Both are far smaller than the +3.51 nats (~33× PPL) the same leak pathway cost on the much larger OpenWebText-scale variant.

• The "Fock paradox" gap analysis below is updated accordingly: 1.20 PPL / 12.4% gap vs the honest MLP baseline (was reported as 1.06 PPL / 11.4% under the leaky architecture).
• A separate, unrelated error was also found and fixed while re-certifying this model: the architecture table below previously reported 17,407,980 total parameters, copy-pasted from the MLP-\(V_\theta\) sibling card. The true count for this SQ3 variant, verified by loading the checkpoint with zero missing/unexpected keys, is 18,194,420 (the SQ3 hypernetwork is larger than the 3-layer MLP it replaces).
• The structured \(V_\theta\) contributions (analytical gradients, explicit attractor centres, landscape compression) are unaffected — only the absolute PPL and parameter-count numbers changed.

For the full root-cause analysis, see Fock-PARFLM_Causal_Leak_Audit_Results.md. For the paired leaky-vs-fixed rerun on this exact configuration, see this repo's causal_leak_fix_verification.md.

Fock-PARFLM v2.1 with Structured V_theta (SQ3 Mixture of Quadratic Wells)

A structured variant of the Fock-PARFLM v2.1 language model in which the MLP scalar potential \(V_\theta\) is replaced by a mixture of K=8 diagonal quadratic wells (SQ3), while retaining the full MLP-based pairwise potential \(V_\phi\) and the Fock register mechanism (16 registers, LIFO stack discipline, reverse channel, now leak-free via prefix_causal_registers=True). This replacement yields:

• Full analytical gradients for \(V_\theta\) — no torch.autograd.grad needed for the scalar potential force, eliminating the second-order computation graph for the \(V_\theta\) component
• Explicit attractor centres — the 8 semantic attractors \(\mu_k(\xi)\) are readable directly from the model parameters, with no gradient-descent extraction required
• Most compressed \(V_\theta\) landscape — with both \(V_\phi\) and Fock registers carrying the force budget, \(V_\theta\) collapses to the flattest bias field observed across all three architectures (mean 0.017, range 16.0)

The trade-off is a 1.20 PPL gap: 10.90 PPL (honest, leak-free) vs the MLP baseline's 9.70 PPL (12.4% excess cross-entropy). This is larger than the PARFLM gap (0.17 PPL) despite even greater landscape compression — the "Fock paradox" discussed below.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Model
• Architecture
• SCAF Causal-Leak Audit (Formal)
• About the SCAF Library
• The Fock Paradox: Maximal Compression, Moderate Gap
• How to Get Started
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation

When to Use This Model

Choose this structured variant over the MLP-based Fock-PARFLM v2.1 when:

Both PPL numbers are from the leak-fixed architecture (prefix_causal_registers=True); see the causal-leak notice above for the fix and its small, honest cost.

Bottom line: for Fock-PARFLM, structured \(V_\theta\) is a viable option when interpretability or analytical-gradient inference is valued — the 1.20 PPL cost is meaningful but the attractor readout and speed gains may justify it. For maximum PPL, use the MLP baseline.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps:
       |
       +-- K-EMA channels: xi^(k)_t = causal_ema(h, alpha_k)   [K_xi=4 channels]
       |
       +-- Structured V_theta (SQ3):
       |     xi_flat = flatten(xi_1..xi_K)                      [K_xi * d = 1024]
       |     V = -tau * logsumexp_k(-E_k/tau + log pi_k)        [K_mix=8 wells]
       |     f_theta = -analytical_grad_h V                     [closed-form]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Fock register pool (v2.1):
       |     M=16 virtual registers with Q/K/V creation gates
       |     LIFO stack discipline, salience decay
       |     Per-register tau and key subspaces
       |     Reverse channel (non-conservative exchange)
       |     *** register state is now per-position (prefix-causal fix); no leak ***
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Damped Stormer-Verlet step: h += (h-h_prev)/(1+dt*gamma) + dt^2*f/(m*(1+dt*gamma))
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

Effective damping. The nominal gamma = 0.30 overstates the true dissipation. The LayerNorm applied after each integration step rescales the hidden state, absorbing most of the velocity decay. The dynamics are therefore heavily underdamped even at this nominal value. Gamma-sweep experiments on the OpenWebText-scale depth-conditioned variant confirm that the effective damping is much smaller than the nominal coefficient.

SCAF Causal-Leak Audit (Formal)

The causal-leak notice above and causal_leak_fix_verification.md document a hand-built verification of the fix. This section adds a formal, tool-generated audit of this exact checkpoint (checkpoint/ckpt_best.pt) using SCAF's standardised probe battery. Verdict: CLEAN, with every control and probe passing:

On the raw PPL numbers inside these JSON files: target_relocation and the legacy probe report standard/honest PPL in the billions (e.g. detail_ppl_standard: 19,317,229,008). This is not a measurement of the model's real quality — it is a byproduct of scoring a tiny, deliberately adversarial 64-target diagnostic slice with the probe's own scoring convention, and is expected to look nothing like the model's real 10.90 PPL validation score reported elsewhere on this card. The number that matters here is the ratio between the standard and honest variants (0.999996, i.e. effectively 1.0x inflation) and the paired difference (-3.8e-06 nats, noise-floor), not either value in isolation.

Tier-B (well-parameter) geometric probes are skipped for this checkpoint (has_vtheta_wells: false) — the structured-quadratic SQ3 potential does not expose the Gaussian-mixture well_parameters() interface Tier-B needs, a known, expected gap for this \(V_\theta\) family rather than an audit failure. Phase-4's register-usage diagnostics (entropy/diversity/creation-gate-strength per layer, verdict MIXED) are a separate, non-leak diagnostic — SCAF's mediation/diagnostic outputs never affect the CLEAN/LEAK verdict — included here for completeness rather than as evidence either way.

The middle-left panel (CATE by distance to cut) is the clearest single signal: a real leak produces a spike concentrated near distance zero that decays with distance (the "shared-register leak" profile SCAF's own documentation describes); here the profile is flat at exactly zero across the entire window, the expected picture for a genuinely causal model. Full raw output: scaf_analysis_results/phase1_scaf_audit.json through phase6_estimation_report.json in this repository.

About the SCAF Library

This audit was produced with SCAF (SemSimula Causal Auditing Framework), a standalone, model-agnostic library that resolves onto a checkpoint by structural (duck) typing — no base class or Protocol required. Its causal audit (on main, used above) detects, sizes, and attributes causal leaks via two complementary probes — future_perturbation (bit-exact: is there a leak at all?) and target_relocation (how many nats of perplexity were unearned?) — backed by three controls (determinism, placebo, positive) that gate the CLEAN verdict, a mediation knockout diagnostic for locating a confirmed leak, scaf.LeakMonitor for continuous training-time monitoring, and a DoWhy/EconML estimand bridge for exact paired-inference ATEs and CATE-by-distance heterogeneity profiles.

SCAF also has a second, independent audit family — a stiffness audit (scaf.StiffnessProbe, on the stiffness_audit branch, scheduled to merge into main) that checks a trained model's numerical integration stability rather than its causal structure. It is not applicable to this checkpoint's causal-audit results above, but is directly relevant to this architecture family: see dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability for a worked example where it identified a structural Störmer–Verlet instability in a sibling (anisotropic-Gaussian) \(V_\theta\) family.

The Fock Paradox: Maximal Compression, Moderate Gap

Across the three SPLM-family architectures tested with structured \(V_\theta\), the Fock-PARFLM exhibits a striking paradox: the flattest \(V_\theta\) landscape yet the second-largest expressivity gap.

Both Fock-PARFLM PPL values (structured and MLP) are now leak-free, from re-trained checkpoints with prefix_causal_registers=True. The SPLM and PARFLM values were always leak-free (no reverse channel). All cross-architecture comparisons in this table are now valid on a like-for-like (honest PPL) basis.

The landscape compression is monotonic (SPLM > PARFLM > Fock-PARFLM), but the expressivity gap is non-monotonic: PARFLM achieves the smallest gap despite a less compressed landscape than Fock-PARFLM. This ranking is unchanged after the leak fix — only the absolute PPL and gap magnitudes moved.

Why? Operating close to the dataset's entropy floor (9.70 PPL for the honest MLP baseline), the Fock model is where the marginal value of each nat of \(V_\theta\) precision is higher. The Fock register mechanism (creation/destruction operators, stack discipline, reverse channel) creates a more structured dynamical regime where even a near-flat \(V_\theta\) must provide fine-grained corrections that the diagonal quadratic parameterisation cannot match. The relationship between landscape compression and expressivity gap is therefore architecture-dependent, modulated by proximity to the entropy floor.

The force from the structured \(V_\theta\) is computed in closed form:

$$f_\theta = -\nabla_h V_\theta = -\sum_{k=1}^{K} q_k(\xi, h) \cdot a_k(\xi) \odot (h - \mu_k(\xi))$$

where \(q_k\) are the softmax responsibilities over the 8 quadratic wells. The \(V_\phi\) force still uses autograd (sparse, over top-k=8 neighbours only), and the Fock forces use their own differentiable computation.

For full derivations, all four structured variants (SQ1--SQ4), landscape compression analysis, attractor basin decoding, and hyperparameter selection strategies, see the companion note: Structured_VTheta_Design_and_Theory.md.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_structured_vtheta import MixtureQuadraticVTheta
from parf.model_structured_vtheta_multixi import StructuredVThetaMultiXiAdapter

# -- Build base model --
config = FockMultiXiPARFConfig(
    vocab_size=50257, d=256, L=8,
    v_hidden=1024, v_depth=3,
    max_len=1024,
    fixed_gamma=0.30, xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True, mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
    fock_version="v2",
    n_registers=16, d_k=64,
    tau_create_init=8.0,
    register_salience_decay=0.5,
    register_salience_threshold=0.01,
    stack_discipline=True,
    reverse_channel=True,
    per_register_tau=True,
    per_register_keys=True,
    ortho_register_init=True,
    prefix_causal_registers=True,  # leak fix — must be True to reproduce this checkpoint
)
model = FockMultiXiPARFLM(config)

# -- Swap in structured V_theta --
K_xi, d = 4, 256
inner = MixtureQuadraticVTheta(d=d, K=8, tau=1.0, xi_d=K_xi * d)
model.V_theta = StructuredVThetaMultiXiAdapter(inner, K=K_xi, d=d)

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-fock-parflm-structured-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                        # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)      # (1, 64, 8, 1024)
    print(f"Attractor centres shape: {centres.shape}")   # 8 basins per token

Available Artifacts

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens; validation: ~140k tokens.

Training Procedure

The base model architecture is identical to the Fock-PARFLM v2.1. The only modification is the \(V_\theta\) replacement: the 3-layer MLP is swapped for the SQ3 mixture at model construction time, before training begins from scratch. The pairwise \(V_\phi\) (competitive structural MLP, hidden=128, top-k=8) and Fock register pool (16 registers, LIFO, reverse channel) are unchanged.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_multixi_structured_vtheta.ipynb --- Colab notebook with structured V_theta arms, GDrive output, checkpointing, and live progress display.

Evaluation Results

TinyStories Validation Perplexity

Both Fock-PARFLM models (MLP and SQ3) were re-trained with prefix_causal_registers=True and independently certified leak-free (bit-exact 0.0 future-perturbation sensitivity). All PPL values in this table are now on a like-for-like, honest basis. Fock-PARFLM (SQ3) still beats both Multi-Xi models, but its edge over the honest MLP-\(V_\theta\) Fock sibling (9.70) is now a 1.20 PPL gap rather than the 1.06 PPL gap reported under the leaky architecture.

V_theta Landscape Statistics

Recomputed on the leak-fixed checkpoint (previously: mean 0.008, std 0.26, range 19.1 on the leaky checkpoint). Both runs show the same near-flat landscape; exact values differ because the fixed run is an independently-trained model with different weights.

Learned Xi-Channel Decay Rates

The final learned alpha values \([\alpha_1, \ldots, \alpha_4] = [0.14, 0.56, 0.79, 0.95]\) are stable and consistent with the SPLM \([0.12, 0.59, 0.84, 0.97]\) and PARFLM \([0.11, 0.55, 0.81, 0.97]\) values, confirming that the causal EMA context structure is invariant to the \(V_\theta\) parameterisation, the presence of \(V_\phi\), and the Fock dynamics.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. This model is a proof-of-concept for structured scalar potentials in Fock-augmented architectures, not a production system.
• TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• English only. No multilingual capability.
• Small scale. 18.2M parameters, 256-dim hidden states.
• No safety training. No RLHF, DPO, or safety filtering has been applied.
• V_phi and Fock forces still use autograd. Only the \(V_\theta\) gradient is analytical; the \(V_\phi\) pairwise force and Fock register forces still require torch.autograd.grad. The overall training speedup is therefore partial (\(V_\phi\) and Fock forces dominate the cost).
• Moderate expressivity gap. The 1.20 PPL gap (12.4% excess CE) is larger than the PARFLM structured V_theta gap (0.17 PPL), reflecting the higher precision demands of the Fock architecture near the dataset's entropy floor.
• Causal leak in the reverse channel — resolved. The Fock reverse channel previously introduced a causal leak by blending each token's content into a global register state shared across all sequence positions within the same integration step. On the OpenWebText-scale variant of the same architecture, a causal probe measured +3.51 nats of NLL inflation (~33× PPL factor) from this pathway. This checkpoint was re-trained with the architectural fix (prefix_causal_registers=True) and certified leak-free via a bit-exact (0.0, float64) future-perturbation probe and honest-vs-standard PPL agreement within noise. The fix cost a modest +0.54 PPL relative to a like-for-like leaky rerun (10.36 → 10.90). See the full audit, this repo's causal_leak_fix_verification.md, and the formal SCAF Causal-Leak Audit section above (verdict CLEAN, all controls and probes passing).

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: NVIDIA A100 40GB (Google Colab)
• Training time: ≈3 hours for the original leaky run (16,000 steps, A2 arm). The re-trained, leak-fixed checkpoint on this page took noticeably longer — checkpoint timestamps for the final segment (steps 14,400→16,000) imply ≈4.7s/step, i.e. a full run on the order of ≈20 hours, a larger relative slowdown than the ≈3× seen on the MLP-\(V_\theta\) sibling (per-position register bookkeeping is a bigger fraction of a lighter, analytical-gradient step).
• Carbon footprint: Estimated less than 10 kg CO2 for the leak-fixed rerun (was <2 kg CO2 for the original leaky run).

Correction (2026-09-27) — integrator name. This model's layer step was described here as a damped Euler step. It is not. The update is h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma)), which carries no velocity state at all: the velocity is the position difference h - h_prev. Undamped this is h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m, i.e. Störmer–Verlet in position form. It is also not velocity-Verlet, which carries an explicit velocity through half-kick/drift/half-kick. The distinction matters because the \(\omega \cdot dt < 2\) stability wall is the Störmer/leapfrog bound and applies to this lineage; the semi-implicit Euler models in the same collection were never subject to it. Only the name was wrong — no measurement on this card changes.

Correction (2026-09-28) — this model is NOT purely conservative. Its config.json sets use_reverse_channel: true, and the reverse channel — the register-to-token path — is not the gradient of anything in \(h\). Later measurement on this architecture puts the deflection of the layer step from the damped \(V_\theta\) geodesic at 1.09 with the reverse channel on and 0.0003 with it off, i.e. the non-gradient term carries most of the step. The conservative-by-construction claim, and the Riemannian capabilities that were derived from it, hold for the conservative sector \(V_\theta + V_\varphi\) and for siblings trained without the reverse channel — not for this checkpoint as shipped. Measurements on this card are unaffected; only the framing was wrong.