Hybrid SPLM (Scalar-Potential + Attention Language Model)

The Hybrid SPLM combines an attention front-end with a scalar-potential refinement back-end in a two-stage architecture. The attention blocks gather global context across positions (what attention does best), then the SPLM blocks refine each position deterministically through a learned energy field (what conservative dynamics does best). At decode time, the SPLM tail costs \(O(d^2)\) per token independent of sequence length -- the FLOP-efficiency hypothesis the hybrid is designed to test.

This model achieves 8.50 PPL on TinyStories, within 0.69 PPL of matched pure attention (7.81), and is the best-performing variant in the Semantic Simulation SPLM family.

Table of Contents

• Model Details
• Architecture
• Why Not a Pure Transformer?
• Geometric Capabilities
• How to Get Started
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation
• Environmental Impact

Model Details

Model Description

The Hybrid SPLM (Variant A) is a two-stage autoregressive language model:

1. Attention stage (k=4 blocks): Standard causal multi-head self-attention blocks with residual connections, producing a contextualised representation.
2. SPLM stage (m=4 steps): Conservative scalar-potential dynamics that refine the attention output through gradient-driven integration steps.

The single causal cumulative-mean context \(\xi\) is re-derived from the detached attention output, preserving the causal-honesty invariant. A single shared \(V_\theta\) drives all integration steps.

• Developed by: Dimitar P. Gueorguiev (Independent Researcher)
• Model type: Hybrid attention + conservative autoregressive language model
• Language: English
• License: CC-BY-4.0

Model Sources

• Paper: Semantic Simulation: A Prescriptive Lagrangian Framework for Efficient Semantic Inference
• Repository: github.com/dimitarpg13/semsimula-paper
• Model source code: notebooks/conservative_arch/hybrid/model_hybrid.py

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   === ATTENTION STAGE (k=4 blocks) ===
   For i = 1..4:
       h = AttnBlock_i(h)             [causal multi-head self-attention + FFN]
       |
   LayerNorm(h)
       |
   === SPLM STAGE (m=4 steps) ===
   xi = causal_cumulative_mean(h.detach())   [leak-safe re-derivation]
       |
   For j = 1..4:
       f = -grad_h V_theta(xi, h)     [conservative force from shared potential]
       v = (v + dt*f/m) / (1 + dt*gamma)
       h = h + dt*v
       LayerNorm(h)
       |
   Logits = h @ E^T                   [tied embeddings]

Effective damping. The learned \(\gamma = 0.166\) overstates the true dissipation. The LayerNorm applied after each SPLM integration step rescales the hidden state, absorbing most of the velocity decay. The dynamics in the SPLM refinement tail are therefore heavily underdamped even at this nominal value. Gamma-sweep experiments on the OpenWebText-scale Fock-PARFLM variant confirm that the effective damping γeff is much smaller than the nominal coefficient across all SPLM-family models.

Key Design Properties

• Best of both worlds: Attention handles global token routing; conservative dynamics handles local refinement.
• FLOP-efficient decoding: At long context \(T \gg 1\), the SPLM tail adds \(O(d^2)\) per token vs \(O(Td)\) for attention with KV-cache. The embed+logits floor limits short-context savings to ~9%, but at \(T \geq 4096\) the hybrid achieves -39% decode FLOPs at PPL parity.
• Helmholtz decomposition: The two-stage architecture realises a learned Helmholtz decomposition -- the attention front-end breaks the conservative gauge, and the SPLM back-end operates in a gauge-fixed space (Section 17b of the paper).

Why Not a Pure Transformer?

The Hybrid SPLM is a two-stage architecture that uses attention only in the front-end (4 blocks) and replaces the remaining computation with scalar-potential gradient dynamics (4 SPLM steps). Unlike a pure Transformer, the SPLM refinement stage has no KV-cache, no FFN towers, and is driven entirely by a single small scalar-potential MLP, \(V_\theta\) — 3-layer, 1024-hidden.

Key structural differences from Transformers:

Note: Because this model retains attention blocks in the front-end, its runtime memory is still \(O(T)\) in sequence length (due to the KV-cache in the attention stage). However, the SPLM tail contributes only \(O(d^2)\) per token, and at long context \(T \geq 4096\) the hybrid achieves -39% decode FLOPs vs matched pure attention. For fully \(O(1)\) inference, see the Multi-Xi SPLM, PARFLM, and Fock-PARFLM.

Geometric Capabilities

Note: This model includes attention blocks in its front-end, breaking the conservative-by-construction guarantee. The full damped Riemannian geometry — layer-dependent Jacobi metric, damped geodesics with friction, energy dissipation anomaly detection, curvature proxy — is available only in the purely conservative variants: Multi-Xi SPLM and Multi-Xi PARFLM. (Fock-PARFLM v2.1 was listed here until 2026-09-28 and has been removed: it ships with the reverse channel enabled, which is not a gradient of anything in \(h\).). A Riemannian Geometry Diagnostic Battery (June 2026) examined the metric and the damping across those models. Scope, added 2026-09-28: both of that battery's headline readings have since been narrowed. The damping figure it fitted was shown to be an artefact of the fit rather than a property of the model, and the residual diagnostic it used cannot establish geodesy at one integration step per layer. What survives is comparative, not absolute. See the companion note for the full damped framework.

However, the SPLM refinement layers in this hybrid architecture still use the same damped Störmer–Verlet integration with (V_\theta) as the PARFLM and Fock-PARFLM variants (Multi-Xi SPLM uses damped semi-implicit Euler; both are explicit schemes and the geometric structure is defined for either). The geometric structure (metric, geodesics, curvature) is well-defined within those layers — the attention front-end only prevents the global conservative guarantee from holding end-to-end.

How to Get Started

# Clone the companion repository for full source code
# git clone https://github.com/dimitarpg13/semsimula-paper.git
# cd semsimula-paper/notebooks/conservative_arch

import torch
import sys
sys.path.insert(0, "hybrid")
sys.path.insert(0, "sarf_mass_variant")
sys.path.insert(0, ".")

from hybrid.model_hybrid import HybridSPLM, HSPLMConfig

config = HSPLMConfig(
    vocab_size=50257,      # GPT-2 BPE
    d=256,
    n_attn=4,
    n_splm=4,
    n_head=4,
    v_hidden=1024,
    v_depth=3,
    max_len=1024,
    block_size=512,
    gamma_init=0.15,
)

model = HybridSPLM(config)
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# Forward pass
x = torch.randint(0, 50257, (1, 64))
logits, loss = model(x, targets=x)

Available Checkpoint

A trained checkpoint (PPL 8.01, 16k steps) is included in this repository:

To load the checkpoint:

from huggingface_hub import hf_hub_download
import torch

# Download checkpoint
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-hybrid-splm",
    filename="checkpoint/model.pt",
)

# Load into model (after creating model as above)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model_state_dict"])
model.eval()

Training Details

Training Data

TinyStories -- tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens; validation: ~140k tokens.

Training Procedure

Training Script

notebooks/conservative_arch/scaleup/train_hybrid_scaleup.py

Evaluation Results

TinyStories Validation Perplexity

The Hybrid SPLM achieves the best PPL in the family (8.50), but uses attention and therefore does not satisfy the conservative-by-construction constraint that the pure SPLM / PARFLM / Fock variants maintain. It serves as the empirical bridge between attention and the fully conservative designs.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. Proof-of-concept for the hybrid conservative/attention architecture, not a production system.
• TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens).
• English only. No multilingual capability.
• Small scale. ~19M parameters, 256-dim hidden states.
• No safety training. No RLHF, DPO, or safety filtering has been applied.

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: NVIDIA A100 40GB (Google Colab)
• Training time: ~4 hours (8,000 steps, 5 seeds)
• Carbon footprint: Estimated < 5 kg CO2

Correction (2026-09-27) — integrator name. This model's layer step was described here as a damped Euler step. It is not. The update is h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma)), which carries no velocity state at all: the velocity is the position difference h - h_prev. Undamped this is h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m, i.e. Störmer–Verlet in position form. It is also not velocity-Verlet, which carries an explicit velocity through half-kick/drift/half-kick. The distinction matters because the \(\omega \cdot dt < 2\) stability wall is the Störmer/leapfrog bound and applies to this lineage; the semi-implicit Euler models in the same collection were never subject to it. Only the name was wrong — no measurement on this card changes.