⚡ SmolLM2-135M-PolyFFN: Orthogonal Chebyshev Polynomial Weight Surgery

SmolLM2-135M-PolyFFN is an algebraically compressed and surgically transformed version of HuggingFaceTB/SmolLM2-135M-Instruct.

All 30 Transformer feed-forward network (FFN / SwiGLU) layers have been mapped into orthogonal Chebyshev polynomial tensor operators using Zero-Backpropagation closed-form algebraic solving on an idempotent spectral manifold (Π² = Π).

• Author & Inventor: Dr. A. Emre ÇETİN (aemre.cetin@gmail.com)
• Official Patent: Protected under USPTO Application No. 64/149,540 (Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers).
• Affiliation: Computational Systems and Cognitive Architectures, Izmir, Turkey
• Core Library: idempotent-poly (pip install idempotent-poly)
• Research Paper: ResearchGate Publication 414060833
• Sister Model (From Scratch): aecetin/PolyFormer-Tiny

🔬 Official Verified Benchmarks

Evaluated on SmolLM2-135M-Instruct across multiple surgical depth configurations:

Architectural Comparison

🔗 Sister Architecture: Ground-Up PolyFormer (From Scratch)

Beyond weight surgery on existing LLMs, orthogonal polynomial tensors have been evaluated as a complete replacement for standard Transformers from scratch:

• Checkpoint: aecetin/PolyFormer-Tiny
• -%46.2 Fewer Parameters (609K vs 1.13M baseline)
• 202.7 tokens/sec real-time generation on CPU
• 93.06% Next-Token Cosine Alignment with standard Transformer
• Overtakes 1.13M model in 300 steps (PPL 1.48 vs 1.50) while locking in permanent 50% inference RAM savings.

📐 Mathematical Formulation

Instead of high-rank multi-matrix projections with intermediate non-linear activations:

$$ \text{FFN}(x) = \text{down_proj}(\text{SiLU}(\text{gate_proj}(x)) \odot \text{up_proj}(x)) $$

We project input representations directly onto orthogonal Chebyshev polynomial basis manifolds:

$$ T_0(x) = 1, \quad T_1(x) = x, \quad T_{k+1}(x) = 2x T_k(x) - T_{k-1}(x) $$

The feed-forward mapping is computed via tensor contraction:

$$ \text{PolyFFN}(x) = \sum_{k=0}^K C_k \cdot T_k(\tilde{x}) + b $$

The coefficient tensor is solved analytically via regularized normal equations:

$$ C^* = (\Phi(X)^T \Phi(X) + \lambda I)^{-1} \Phi(X)^T Y $$

and projected onto the idempotent subspace:

$$ \Pi = V_r V_r^T, \quad \Pi^2 = \Pi $$

💻 Quickstart & Inference

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from idempotent_poly import ChebyshevPolyFFN

# Load model and tokenizer
model_id = "HuggingFaceTB/SmolLM2-135M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32)

# Surgically replace Layer 15 with ChebyshevPolyFFN in 56 ms
target_layer = model.model.layers[15]
calib_tokens = tokenizer(["Mathematics and artificial intelligence thrive on idempotent manifolds."], return_tensors="pt").input_ids

captured = {}
def hook_fn(m, inp, out):
    captured["X"] = inp[0].detach().reshape(-1, model.config.hidden_size)
    captured["Y"] = out.detach().reshape(-1, model.config.hidden_size)

h = target_layer.mlp.register_forward_hook(hook_fn)
with torch.no_grad():
    model(calib_tokens)
h.remove()

poly_ffn = ChebyshevPolyFFN(d_model=model.config.hidden_size, degree=3)
poly_ffn.fit_algebraic(captured["X"], captured["Y"], l2_reg=1e-2, rank_ratio=0.95)
target_layer.mlp = poly_ffn

# Generate text
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=25, do_sample=False)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Output: The capital of France is Paris. Paris is a city that is known for its historical landmarks

📜 Citation & Intellectual Property

This work and its underlying mathematical architectures are protected under United States Patent Law:

@patent{cetin2026orthogonalpoly,
  title={Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers},
  author={Dr. Ahmet Emre {\c{C}}etin},
  year={2026},
  month={September},
  note={U.S. Provisional Patent Application No. 64/149,540, Filed at USPTO}
}