⚡ PolyFormer-Tiny: Orthogonal Chebyshev Polynomial LM (Trained From Scratch)

PolyFormer-Tiny is a native, non-linear neural language model architecture trained completely from scratch without standard multi-layer perceptrons (MLP / SwiGLU).

Instead of conventional matrix projections with static activation functions, PolyFormer replaces all feed-forward network (FFN) blocks with orthogonal Chebyshev polynomial tensor operators and attention heads with idempotent polynomial kernels on an idempotent manifold (Π² = Π).

• Author & Inventor: Dr. A. Emre ÇETİN (aemre.cetin@gmail.com)
• Official Patent: Protected under USPTO Application No. 64/149,540 (Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers).
• Affiliation: Computational Systems and Cognitive Architectures, Izmir, Turkey
• Core Library: idempotent-poly (pip install idempotent-poly)
• Research Paper: Section IV & V: PolyFormer Architecture & Benchmark - ResearchGate Publication 414060833
• Sister Model (In-Situ Surgery): aecetin/SmolLM2-135M-PolyFFN

🔬 Official Verified Benchmarks: Inference & Architecture

Direct side-by-side benchmark evaluated against an architecturally identical Standard Transformer baseline:

⚖️ Semantic & Output Alignment (Standard vs PolyFormer)

To rigorously verify that halving parameters does not degrade representational fidelity, both models were evaluated under identical prompt conditions:

1. Vector Alignment (Next-Token Cosine Similarity)

2. Convergence Summary

3. 📈 Compute-Optimal Scaling: Catching Up to Double-Sized Baselines

A central question in compact non-linear design is whether parameter-reduced models hit an insurmountable capacity wall. To empirically verify this, we tracked PolyFormer's learning curve across extended optimization horizons:

Key Theoretical Takeaway: Because orthogonal Chebyshev tensor contractions span a richer non-linear basis than static linear GEMMs, modest compute extension allows a 46.2% smaller model to fully exceed double-sized baseline capacities while permanently locking in 50% inference memory and VRAM savings.

📐 Mathematical Formulation

1. Orthogonal Chebyshev Basis Recursion

Input representations are normalized onto the Chebyshev domain $[-1, 1]$:

$$ \tilde{x} = \tanh(x) \in [-1, 1] $$

and projected onto orthogonal Chebyshev polynomials of the first kind:

$$ T_0(x) = 1, \quad T_1(x) = x, \quad T_{k+1}(x) = 2x T_k(x) - T_{k-1}(x) $$

2. Chebyshev Tensor FFN

Instead of 3 separate high-rank GEMM matrices (Gate, Up, Down projections), the feed-forward mapping is computed directly via 3D tensor contraction:

$$ \text{PolyFFN}(x) = \sum_{k=0}^K C_k \cdot T_k(\tilde{x}) + b $$

where the orthogonal coefficient tensor is parameterized as:

$$ C \in \mathbb{R}^{(K+1) \times d_{\text{in}} \times d_{\text{out}}} $$

3. Idempotent Chebyshev Attention Kernel

Attention scores pass through an orthogonal polynomial kernel filter:

$$ S = \frac{QK^T}{\sqrt{d_k}} $$

$$ \text{Attn}(Q, K, V) = \text{softmax}\left( \sum_{k=0}^K w_k T_k\left(\tanh(S)\right) \right) V $$

💻 Quickstart & Inference

Using Transformers (AutoModelForCausalLM)

import torch
from transformers import AutoConfig, AutoModelForCausalLM

# Load model directly from Hugging Face Hub
repo_id = "aecetin/PolyFormer-Tiny"
config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, config=config, trust_remote_code=True)

# Encode prompt using byte-level tokens
prompt = "Mathematics, artificial intelligence, and geometric deep learning"
input_ids = torch.tensor([list(prompt.encode("utf-8"))], dtype=torch.long)

# Autoregressive generation
with torch.no_grad():
    output_tokens = model.generate_tokens(input_ids, max_new_tokens=48, temperature=0.7)

generated_text = bytes(output_tokens[0].tolist()).decode("utf-8", errors="ignore")
print(generated_text)

Using Core Library (idempotent-poly)

pip install idempotent-poly

import torch
from idempotent_poly.polyformer import PolyFormerBlock

# Instantiate native PolyFormer block
block = PolyFormerBlock(d_model=128, n_heads=4, poly_degree=3)

# Test adaptive fixed-point early halting
x = torch.randn(1, 16, 128)
output, steps_taken = block.forward_adaptive(x, max_iters=5, tol=0.02)
print(f"Fixed-point reached in {steps_taken} adaptive iteration steps!")

📜 Citation & Intellectual Property

This work and its underlying mathematical architectures are protected under United States Patent Law:

@patent{cetin2026orthogonalpoly,
  title={Hardware-Accelerated Orthogonal Polynomial Tensor Operators, Zero-Backpropagation Closed-Form Algebraic Solvers, and In-Situ Weight Surgery for Deep Neural Networks and Transformers},
  author={Dr. Ahmet Emre {\c{C}}etin},
  year={2026},
  month={September},
  note={U.S. Provisional Patent Application No. 64/149,540, Filed at USPTO}
}