Spartacus-1B-Instruct — Causal Monoid Language Model

A 1.3B parameter language model that replaces softmax attention with causal monoid state compression, achieving O(1) time per token and O(1) memory at inference — regardless of sequence length.

SFT Training Curves

Loss	Accuracy

Core Mechanism

Key Properties

The Monoid Recurrence

Standard attention computes:

o_t = Σ_{i≤t} softmax(q_t · k_i) v_i    — requires O(T) KV-cache

Monoid attention compresses the entire causal history into a fixed-size state matrix S_t per head:

S_t = diag(α_t) · S_{t-1} + k_t ⊗ v_t     — vector decay monoid recurrence
o_t = q_t · S_t                              — state readout

This is a monoid because the binary operator (α, S) ⊕ (β, X) = (α·β, diag(β)·S + X) is associative, enabling O(T) parallel prefix scan for training and O(1) sequential update for inference.

Vector Decay — Per-Dimension Memory Lifetimes

Unlike scalar decay (one α per head), Spartacus uses vector decay: each dimension of the d-vector has its own independent decay rate α_t[i] ∈ (0, 1):

S_t[i,j] = α_t[i] · S_{t-1}[i,j] + k_t[i] · v_t[j]

This allows different feature dimensions to specialize:

• Fast-decaying dimensions (α ≈ 0) — local syntax, punctuation, function words
• Slow-decaying dimensions (α ≈ 1) — entity memory, topic tracking, long-range facts

The decay gate uses Sigmoid activation:

α_t = σ(W·x_t + b)

Attention Mask — Padding-Aware Recurrence

The monoid recurrence correctly handles attention_mask for padded batches (e.g., left-padding during generate()). For PAD positions (mask=0):

α = α * mask + (1 - mask)  →  α = 1  (preserve state unchanged)
k = k * mask, v = v * mask →  kv = 0 (no information injected)

Net effect: S_t = 1·S_{t-1} + 0 = S_{t-1} — PAD acts as the monoid identity element, completely invisible to the recurrence. This ensures identical outputs whether inputs are padded or not.

Design Choices

• SiLU-activated keys: k = SiLU(k_proj(x)) ensures non-negative keys, making the state matrix S positive semi-definite (PSD). This prevents "feature erasure" where one token's contribution cancels another's
• QK-Norm: RMSNorm on both q and k before readout, stabilizing the scale of q·S when the state matrix accumulates many outer products
• Output Norm: RMSNorm on the readout o after q·S, further stabilizing scale before gating
• Output Gate: gate = SiLU(gate_proj(x)), modulates the multi-head readout before o_proj (similar to GLA/RetNet). Lets the model suppress or amplify specific head outputs conditioned on the current input
• Sigmoid decay gate: Ensures α ∈ (0, 1) by construction — allows near-perfect memory (α→1) while preventing state explosion (α>1). Bias initialized to 3.0 so σ(3)≈0.95, starting in high-retention mode
• Learnable h0: The initial state S₀ = h0 is a learnable parameter (zero-initialized), acting as a compressed "system prompt"
• Log-space decay in scan: The parallel prefix scan works in log-space log(α) to avoid numerical underflow when computing cumulative products over long sequences

Three Forward Paths

Model Details

Benchmarks (0-shot)

Comparison with ~1B Baselines (acc_norm, 0-shot)

Spartacus achieves competitive performance with sub-quadratic models (Mamba, RWKV) while maintaining O(1) inference time and memory per token. Scores marked with ~ are approximate community-reported values.

Parallel Scan Implementation

The monoid_scan_cuda.py module provides a Triton JIT-compiled parallel prefix scan for the vector-decay monoid:

• Grid: (B*H*D_k, ceil(D_v/BLOCK_DV)) — one program per state matrix row
• Forward: Sequential scan along T per row, parallelized across all (batch, head, d_k) dimensions
• Backward: Reverse-order adjoint scan with per-row D_v reduction (minimal atomic_add)
• Fallback: Pure PyTorch sequential scan for CPU/MPS
• Auto-dispatch: CUDA → Triton kernel, otherwise → PyTorch fallback

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "NoesisLab/Spartacus-1B-Instruct",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("NoesisLab/Spartacus-1B-Instruct")

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

File Structure

MonoidForCausalLM.py       # Model architecture (MonoidConfig, MonoidAttention, MonoidForCausalLM)
monoid_scan_cuda.py        # Triton JIT parallel prefix scan (vector decay) + PyTorch fallback
model.safetensors          # Model weights (bfloat16)
config.json                # Model configuration
tokenizer.json             # Llama-3.2 tokenizer
ARCH.png                   # Core mechanism diagram (monoid recurrence + parallel scan)
ACC_SPAR.png               # SFT accuracy curve
LOSS_SPAR.png              # SFT loss curve

Citation

@software{spartacus2025,
  title={Spartacus: Causal Monoid Language Model with O(1) Inference},
  author={NoesisLab},
  year={2025},
  url={https://huggingface.co/NoesisLab/Spartacus-1B-Instruct},
  description={Replaces softmax attention with vector-decay monoid state compression for constant-time, constant-memory autoregressive generation}
}

License

Apache 2.0