Fock-PARFLM v2.1 Anisotropic Gaussian V_theta + Fock Regularisation — Verlet Instability at Full-Run Scale (OpenWebText, d=384)

This repository holds the two full-scale training attempts ( \(\gamma = 0.10\) and \(\gamma = 0.30\)) that followed the d=384 gamma sweep, together with a complete SCAF causal-leak and numerical-stiffness audit of both. Neither run is a successfully completed model. Both were targeting 100,000 steps; both instead entered a persistent cycle of gradient spikes and watchdog reloads well short of that target, and neither improved further once the cycle began. This repository exists to document why, with checkpoints and audit evidence retained specifically so the failure mode can be studied.

The headline finding, from the SCAF stiffness audit (Phase 7/7b/7c):

The model's own explicit Störmer–Verlet integrator is running past its analytical stability boundary. The anisotropic-Gaussian \(V_\theta\)'s off-diagonal (rank-4 low-rank) curvature pushes the effective per-token stiffness \(\omega \Delta t\) above the Verlet stability bound of 2 at every single audited checkpoint in both runs — while a naive diagonal-only proxy for that same curvature stays under 1 throughout and would have reported no problem at all. This is a structural property of the trained potential, not a training-recipe bug, and it is what motivated replacing Verlet with a CfC + BAOAB propagator (unconditionally stable with respect to this curvature by construction) as this architecture's default integrator going forward.

Separately, both checkpoint sets pass SCAF's Phase-1 causal-leak audit cleanly — the instability documented here is a numerical-integration problem, not a causal-leak regression.

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Repository
• Architecture
• About the SCAF Library
• Training Runs: Two Damping Coefficients, Two Stalled Runs
• SCAF Causal-Leak Audit: Clean
• SCAF Stiffness Audit: The Instability Mechanism
• Why the Diagonal Proxy Misses It
• Trajectory-Fidelity Cross-Check (Phase 7c)
• What This Motivated: CfC + BAOAB
• How to Get Started
• Available Artifacts
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

When to Use This Repository

Use this repository if you want to:

• Study a real, reproducible instance of explicit-symplectic-integrator instability in a trained conservative language model, including the exact checkpoints, training logs, and audit artifacts involved.
• Reproduce the SCAF stiffness audit methodology (diagonal proxy, Weyl-inequality upper bound, native-trajectory cross-check) against these or similar checkpoints.
• Compare the anisotropic-Gaussian \(V_\theta\) family's instability signature across two damping coefficients ( \(\gamma = 0.10\) and \(0.30\) ) trained on identical architecture and data.

Do not use this repository if you want a well-trained OpenWebText language model: both checkpoint sets stalled between 7% and 17% of their 100,000-step target, and neither reflects converged behaviour. For a well-trained checkpoint from an earlier, isotropic- \(V_\theta\) version of this architecture, see semsimula-fock-parflm-depthcond-vtheta-openwebtext (27.23 PPL, 250K steps). The anisotropic- \(V_\theta\) + CfC-BAOAB successor run that this repository's finding motivated is expected to supersede both once it completes.

Architecture

Identical Fock-PARFLM v2.1 scaffold to the d=384 gamma sweep that selected these damping coefficients — same width, depth, \(V_\theta\) family, and Fock register configuration, now run to (attempted) full length instead of a 3,000-step sweep candidate:

Input tokens x_1, ..., x_T
       |
   Untied token embedding E[x] + learned positional P[t]
       |
   For each of L=16 damped Störmer–Verlet integration steps (shared force field):
       |
       +-- K=5 causal-EMA context channels:
       |     xi^(m)_t = causal_ema(h, alpha_m)      [horizons ~2 .. ~200 tokens]
       |
       +-- Depth-conditioned multi-context V_theta (Anisotropic Gaussian):
       |     xi_g^(m) = xi^(m) + e_g^(m)                    [per-layer depth code]
       |     diff_k^m = h - mu_k^m(xi_g^(m))
       |     V_m = -sum_k w_k^m exp(-0.5[a_k^m . diff_k^{m2} + ||B_k^{mT} diff_k^m||^2])
       |     V_theta = sum_m V_m(xi_g^(m), h)               [5 contexts, 40 wells total]
       |     f_theta = -analytical_grad_h V_theta            [closed-form, bounded]
       |
       +-- Sparse pairwise V_phi (structural-competitive, 4 heads):
       |     top-k=16 past tokens per query (Gumbel routing)
       |     f_phi = -grad_h V_phi(h_t, h_s)                [autograd, sparse]
       |
       +-- Fock register pool (v2, 32 registers):
       |     M=32 virtual registers, Q/K/V creation gates, d_k=64
       |     LIFO stack discipline, per-register tau/keys, ortho init
       |     register repulsion (Gram penalty, lambda=0.05)
       |     reverse channel (per-layer, stabilised, pre-LN, soft-norm, warmup 4000)
       |     prefix-causal (leak-free by construction)
       |     f_fock = creation + destruction + exchange forces
       |
       +-- Total force: f = f_theta + f_phi + f_fock
       |
       +-- Stormer-Verlet step: h += (h-h_prev)/(1+dt*gamma) + dt^2*f/(m*(1+dt*gamma))
       |
       +-- LayerNorm(h)
       |
   Logits = h @ W_out^T + b_out                              [UNTIED W_out]

Auxiliary training-only loss term (not part of the forward pass above):
   L_fock_coupling = -lambda_fock * sum_k log(alpha_k + eps)   [log-barrier on xi coupling]

About the SCAF Library

Every audit result quoted in this card comes from SCAF (SemSimula Causal Auditing Framework), a standalone, model-agnostic library that resolves onto a checkpoint by structural (duck) typing — no base class or Protocol required — and offers two independent audit families:

• Causal audit (on main): detects, sizes, and attributes causal leaks — any path by which a token at position s influences a prediction at an earlier position t < s, which would otherwise report a perplexity the model did not earn. Two probes ask deliberately different questions: future_perturbation (bit-exact — is there a leak at all?) and target_relocation (how many nats of perplexity were unearned?), backed by three controls (determinism, placebo, positive) that a verdict of CLEAN cannot be issued without. A mediation diagnostic then attributes a confirmed leak to the responsible component by knockout, and scaf.LeakMonitor runs the same probes continuously during training, since leak channels are often gated by a scale initialised at zero and open gradually as training proceeds. A build_leak_frame / estimate_leak bridge hands the same intervention to DoWhy/EconML for exact paired-inference ATEs, refutation tests, and CATE-by-distance heterogeneity profiles via a causal forest. This is what certified both this repository's checkpoint sets CLEAN in SCAF Causal-Leak Audit: Clean.
• Stiffness audit (scaf.StiffnessProbe, currently on the stiffness_audit branch, scheduled to merge into main alongside SCAF's other active branches): audits a trained model's numerical integration stability rather than its causal structure, via the diagonal-proxy / Weyl-inequality-upper-bound / native-trajectory-cross-check methodology used throughout this card's stiffness audit section. It is trajectory-faithful by construction (it drives the model through InterventableModel.batch_logits_with_trajectory rather than re-deriving a trajectory by hand), reports block-bootstrap 95% CIs on every fraction-unstable statistic, and skips loudly rather than silently if a model does not expose the harmonic_terms()/mass() hooks it needs.

Full design rationale and worked examples for both audit families: docs/Stiffness_Audit_Requirements_and_Design.md and docs/Example_Stiffness_Audit_OWT_g0.1_Anisotropic_Gaussian.md (both on the stiffness_audit branch as of this card's creation; once merged, the same paths resolve on main).

Training Runs: Two Damping Coefficients, Two Stalled Runs

Both runs used identical architecture, data, and optimisation recipe, differing only in \(\gamma_{\text{train}}\), the damping coefficient selected by the gamma sweep ( \(\gamma = 0.10\) was the sweep winner; \(\gamma = 0.30\) was audited as a second candidate on the same architecture to see whether heavier damping alone would suppress the instability). Neither did:

Both runs show the same qualitative signature: PPL improves normally up to the first watchdog reload cluster, then stops improving — every checkpoint saved after the reloads begin is worse than the pre-cluster best, and the run never recovers a new best afterward within the logged window. Heavier damping ( \(\gamma = 0.30\) ) delayed the first reload (step 7,124 vs. 8,925 — comparable, if anything slightly earlier) and reduced the reload count (5 vs. 15) but did not eliminate the pattern, which is consistent with a structural integrator problem rather than an under-damping problem (see below).

SCAF Causal-Leak Audit: Clean

Both checkpoint sets were run through SCAF's Phase-1 causal-leak audit (FockAdapter, future_perturbation and target_relocation probes, plus determinism/placebo/positive controls). Verdict: CLEAN for both, at every audited checkpoint:

Both checkpoint sets were trained natively with prefix_causal_registers=True from step 0, so this is the expected result, not a new fix — it confirms the instability documented below is purely a numerical-integration problem, with no interaction with the register-leak fix from earlier in this model family. Full audit output: results/gamma_0.10/scaf_analysis_results/phase1_scaf_audit.json and results/gamma_0.30/scaf_analysis_results/phase1_scaf_audit.json.

SCAF Stiffness Audit: The Instability Mechanism

The anisotropic-Gaussian \(V_\theta\)'s force is a sum of Gaussian bumps with non-diagonal curvature:

$$ f(h) = -\sum_k g_k(h) P_k (h - \mu_k), \qquad g_k(h) = w_k \exp\left(-\tfrac{1}{2} \mathrm{diff}_k^\top P_k \mathrm{diff}_k\right) > 0, \qquad P_k = \mathrm{diag}(a_k) + B_k B_k^\top, $$

where \(B_k \in \mathbb{R}^{384 \times 4}\) is the learned low-rank factor giving each well its off-axis curvature. Near a well, this is locally a spring with stiffness matrix \(\sum_k g_k P_k\), and the explicit Störmer–Verlet integrator that this model was trained with is a second-order symplectic method whose stability for a harmonic oscillator with frequency \(\omega\) requires

$$ \omega \Delta t < 2 . $$

Above this bound the discrete update's amplification factor leaves the unit circle and every subsequent step amplifies the deviation instead of oscillating around it — trading a bounded, physically faithful trajectory for one that diverges geometrically, which is exactly the gradient-spike-then-reload signature seen in both training logs.

SCAF's stiffness audit (Phase 7) samples the trained model's own hidden-state trajectory and computes \(\omega \Delta t\) at each sampled (token, layer) position, in two ways:

1. Diagonal proxy (k_diag): keeps only the diagonal of \(\sum_k g_k P_k\), the quantity the CfC+BAOAB propagator (see below) can integrate exactly in closed form.
2. Weyl-inequality upper bound (Phase 7b): applies Weyl's inequality twice — once per well ( \(\lambda_{\max}(P_k) \le \max_i a_k[i] + \sigma_{\max}(B_k)^2\) ) and once across wells — to certify an upper bound on the true, any-direction top eigenvalue of \(\sum_k g_k P_k\), not just its diagonal.

At every one of the nine audited checkpoints across both runs, Weyl max clears the stability bound of 2 (range 2.53-4.25), while diag max never even approaches 1. Statistically, Weyl frac(>2) — the fraction of all sampled (token, layer) positions where the Weyl bound itself exceeds 2 — is consistently 2.7%-6.2% of positions, not a rare tail event: a non-trivial share of the model's own forward pass is operating past this integrator's stability boundary at every checkpoint examined.

Why the Diagonal Proxy Misses It

A synthetic verification (500 trials of random 8-well anisotropic mixtures at d=16, rank=4, checked against the exact eigendecomposition of the true effective matrix) confirms this is a real property of the inequality, not a fluke of these two checkpoints: k_diag's per-dimension maximum underestimated the true top eigenvalue in all 500/500 trials (mean gap 3.77, on a typical eigenvalue scale of 5-18), while the Weyl bound never once fell below the true value. The rank-4 low-rank correction \(B_k B_k^\top\) can concentrate curvature along a direction that is not axis-aligned, and a purely diagonal statistic is structurally blind to exactly that direction — which is precisely the direction the model's own learned wells are using. Full derivation and the synthetic verification plot: Weyl-inequality upper bound section of the SCAF stiffness-audit design doc.

Trajectory-Fidelity Cross-Check (Phase 7c)

Phase 7 samples the audited trajectory under a forced baoab_cfc integrator regardless of a checkpoint's own native integrator, which could in principle change which curvature the audit samples. Phase 7c closes this caveat by re-running each checkpoint under its own native Verlet trajectory instead, for both runs:

Weyl frac(>2) agrees between forced and native trajectories to within 0.02 percentage points at every single checkpoint in both runs, and Weyl max stays clear of the stability bound in both modes at every checkpoint too. The forced-vs-native discrepancy is negligible next to the trend it is being asked to corroborate — the trajectory-substitution caveat is closed for both damping coefficients, by their own checkpoints, without needing extra training.

What This Motivated: CfC + BAOAB

This evidence — a structural curvature violation of the Verlet stability bound present at every audited checkpoint, in both damping regimes, confirmed independent of the audit's own trajectory choice — is what motivated replacing Störmer–Verlet with a Closed-form Continuous-time (CfC) + BAOAB propagator for this architecture going forward. CfC integrates the harmonic component of the force in closed form (via torch.sinc-based exact-flow update rather than an explicit finite-difference step), which is unconditionally stable with respect to this curvature by construction rather than conditionally stable below \(\omega \Delta t = 2\).

This is not a free lunch: the CfC+BAOAB propagator costs roughly 2.4x Störmer–Verlet's per-step wall-clock time on the same GPU. An early CfC+BAOAB run also hit its own regression — a grad=nan traced to torch.sqrt's infinite derivative at exactly k_diag/m == 0 inside cfc_substep — which was fixed with an _OMEGA_SQ_FLOOR = 1e-12 clamp and confirmed to produce zero NaNs across a full resumed training run. The CfC+BAOAB run is in progress as of this card's creation and will be published as its own model card once complete; see the deep-dive writeup for the full implementation: PyTorch_Implementation_of_CfC_BAOAB_in_Fock-PARFLM.md.

How to Get Started

import math, torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_fock_parf_multixi import FockMultiXiPARFLM, FockMultiXiPARFConfig
from parf.model_aniso_gaussian_vtheta import AnisotropicDepthConditionedGaussianVTheta, install_aniso_depth_routing
from huggingface_hub import hf_hub_download

REPO = "dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability"
GAMMA = "0.10"  # also available: "0.30"
STEP = 10000    # the canonical best checkpoint for either gamma

logfreq_path = hf_hub_download(repo_id=REPO, filename=f"results/gamma_{GAMMA}/logfreq_surprisal_openwebtext.npy")

config = FockMultiXiPARFConfig(
    vocab_size=50257, d=384, max_len=1024, L=16,
    mass_mode="logfreq", logfreq_path=logfreq_path,
    init_gamma=1.0, fixed_gamma=float(GAMMA),
    xi_channels=5, xi_alpha_inits=[0.50, 0.75, 0.95, 0.99, 0.995],
    xi_learnable=True, xi_alpha_init_mode="explicit",
    fock_version="v2", n_registers=32,
    reverse_channel=True, reverse_channel_stable=True, reverse_channel_pre_ln=True,
    reverse_channel_soft_norm=True, reverse_channel_warmup_steps=4000, reverse_channel_per_layer=True,
    register_repulsion=True, register_repulsion_coeff=0.05,
    prefix_causal_registers=True,
    v_phi_kind="structural_competitive", v_phi_n_heads=4, v_phi_d_type=32, v_phi_d_angle=16, top_k=16,
    use_output_bias=True, tie_embeddings=False,
    per_register_tau=True, per_register_keys=True, ortho_register_init=True,
)
model = FockMultiXiPARFLM(config)

model.V_theta = AnisotropicDepthConditionedGaussianVTheta(
    d=384, K=8, n_ctx=5, n_layers=16, rank=4,
    w_scale=1.0,
    init_log_precision=-math.log(384),
    precision_max=2.0 / 384,
    code_init_std=0.02,
)
install_aniso_depth_routing(model)

ckpt_path = hf_hub_download(repo_id=REPO, filename=f"checkpoints/gamma_{GAMMA}/ckpt_step{STEP}_best.pt")
state = torch.load(ckpt_path, map_location="cpu", weights_only=False)
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")   # 77,082,733
print(f"gamma={state['gamma']}  step={state['step']:,}  val_ppl={state['val_ppl']:.2f}")

Available Artifacts

Training Details

Training Data

OpenWebText, tokenized with GPT-2 BPE (vocab 50257), evaluated on a held-out 2M-token validation slice with no train/val overlap. Both runs targeted 100,000 steps and were early-terminated (in the sense of "checkpoints stopped being retained as _best") by the instability documented above, at step 17,100 (gamma=0.10) and step 11,254 (gamma=0.30).

Training Procedure

Causal-Leak Verification

See SCAF Causal-Leak Audit: Clean above — both runs pass with verdict CLEAN.

Training Script

notebooks/conservative_arch/scaleup/colab_fock_aniso_gaussian_fockreg_openwebtext.ipynb (companion repo). SCAF audit performed with notebooks/conservative_arch/scaleup/debug/scaf_checkpoint_analysis.ipynb (same repo).

Evaluation Results

OpenWebText Validation Perplexity (best retained checkpoint per run)

Neither of these is comparable to the fully trained OpenWebText checkpoints elsewhere in this family (e.g. 27.23 PPL after 250K steps for the isotropic flagship) — both runs stalled at 10-17% of their 100,000-step target due to the instability documented above.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Not a final trained model — this is a diagnostic repository. Both checkpoint sets stalled at 10-17% of their 100,000-step target and were never intended to produce fluent text; use them to study the instability, not for generation or downstream use.
• PPL not comparable to converged checkpoints elsewhere in this family. See the caveat under Evaluation Results.
• Structural Verlet instability, not a training-recipe bug. The Weyl-bound curvature violation is present at every audited checkpoint across two different damping coefficients and is confirmed independent of the audit's trajectory-substitution choice (Phase 7c); it should be expected to recur in any future Störmer–Verlet-integrated run of this exact \(V_\theta\) family unless the curvature itself is constrained (e.g. by capping the rank-4 factor's spectral norm) or the integrator is replaced, as this repository's finding motivated.
• Weyl bound is an upper bound, not the exact top eigenvalue. A synthetic check found a mean overshoot of 3.77 on eigenvalues typically in the 5-18 range — the reported Weyl frac(>2) values (2.7%-6.2%) are a ceiling on how much of each run crosses the stability line via the off-diagonal direction, not an exact measurement. The exact-eigenvalue extension (torch.linalg.eigvalsh on the small rank-4-derived matrix) is a documented follow-up, not yet run against these checkpoints.
• No causal-leak issue. Both checkpoint sets were trained natively with prefix_causal_registers=True; see SCAF Causal-Leak Audit: Clean.
• OpenWebText only, English only. No instruction tuning, no RLHF/DPO, no safety filtering.
• Single-seed runs. Each of the two damping coefficients is one run; no seed-variance estimate is available for either the instability onset step or its severity.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Companion code repository:
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: 1x NVIDIA H100/A100 80GB (Google Colab)
• Training: 17,100 + 11,254 = 28,354 total logged training steps across both runs, both terminated by the instability documented here rather than reaching their 100,000-step target
• Audit: SCAF Phase 1/7/7b/7c analysis, inference-only, on 9 checkpoints total across both runs
• Carbon footprint: small; single-GPU research runs, estimated on the order of a few kg CO2

Correction (2026-09-27) — "velocity-Verlet" was the wrong name. This lineage's layer step carries no velocity state: the velocity is the position difference h - h_prev, and the update is h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma)), which undamped is h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m. That is Störmer–Verlet in position form. Velocity-Verlet is the variant that does carry an explicit velocity, through half-kick/drift/half-kick, and was never used here. The \(\omega \cdot dt < 2\) stability bound is the Störmer/leapfrog bound and is unaffected by the correction, as is every measurement on this card.