DiscoverLM-70M

A 69M parameter causal language model built on the Mixture-of-Attentions (MoA) architecture — distance-based metric attention that respects the triangle inequality by construction, not approximation.

Every attention head operates in a proper metric space. The geometry is enforced, not hoped for.

What Makes This Different

Standard transformers compute attention as a dot product: Q·Kᵀ. This has no geometric meaning — it's a bilinear form, not a distance. Two tokens can be "close" by dot product while violating basic metric properties.

MoA replaces this with negative squared distance under a learned diagonal Mahalanobis metric, then enforces the triangle inequality through a regularizer over random triples sampled during training. The result: attention weights reflect actual geometric proximity in a space where d(a,c) ≤ d(a,b) + d(b,c) holds.

This isn't a constraint that fights the model. It's structure the model uses.

Architecture

Input → Token Embedding (48K vocab, custom tokenizer)
  │
  ▼
┌──────────────────────────────────────────────────┐
│              MoA Block × 4                       │
│                                                  │
│  ┌─────────┐ ┌──────────┐ ┌────────┐ ┌────────┐ │
│  │  Local   │ │  Global  │ │Channel │ │  MQA   │ │
│  │  Conv    │ │  Metric  │ │  Mix   │ │ Metric │ │
│  │         │ │ (64 heads)│ │        │ │(64 Q)  │ │
│  └────┬────┘ └────┬─────┘ └───┬────┘ └───┬────┘ │
│       └──────┬────┴───────────┴───────────┘      │
│              ▼                                   │
│     Feature Gates + Token Router (top-2)         │
│              ▼                                   │
│        Residual + DropPath                       │
└──────────────────────┬───────────────────────────┘
                       ▼
         HyperFFN (SwiGLU + CausalConv + LowRank)
                       ▼
                   LayerNorm
                       ▼
┌──────────────────────────────────────────────────┐
│            MoA Language Model Head               │
│  (same 4-path mixture → SwiGLU → tied vocab)    │
└──────────────────────┬───────────────────────────┘
                       ▼
                 Logits (48,000)

Core Components

Metric Attention. Queries attend to keys via learned Mahalanobis distance. Each of 64 heads has an 8-dimensional head space with its own diagonal scaling, learnable ball origin, and adaptive radius for sparse pruning. Pairs outside the ball are masked before softmax.

Mixture-of-Attentions Routing. Four parallel paths per token — local depthwise convolution, full multi-head metric attention, gated channel mixing, and multi-query metric attention. A learned router selects top-2 paths per token position. Feature gates scale each path's output before mixing.

BlackHoleRoPE. Rotary position encoding with learned phase perturbations from a compact Fourier basis. Q/K rotations stay unitary. V amplitudes get bounded energy gating clamped to [0.5, 2.0] with optional discrepancy-state modulation.

HyperFFN. Three-branch feedforward: SwiGLU channel MLP, causal depthwise separable convolution, and gated low-rank bottleneck — routed per-token with top-2 sparse selection.

MoA LM Head. The vocabulary projection runs its own mixture-of-attentions (32 heads, head_dim=16) before projecting to logits through a SwiGLU transform. Weight-tied to the input embedding.

Parameter Budget

vs Standard Transformers

Training

Data

Hyperparameters

Results

Best single step: 393 — loss 0.599, token accuracy 88.4%

Loss variance halved across training (σ: 0.291 → 0.142), indicating the mixture-of-attentions learned stable routing preferences as training progressed.

Configuration

{
  "dim": 512,
  "num_layers": 4,
  "attn_heads": 64,
  "mqa_q_heads": 64,
  "lm_attn_heads": 32,
  "lm_mqa_q_heads": 32,
  "metric": "maha_diag",
  "vocab_size": 48000,
  "max_position_embeddings": 1024,
  "ffn_hidden": 1536,
  "mixer_hidden": 768,
  "n_branches": 3,
  "router_topk": 2,
  "use_balls": true,
  "radius_init": 3.5,
  "ti_reg_weight": 0.01,
  "ti_reg_samples": 64,
  "energy_amplification": 9.87,
  "theta_base": 10000.0,
  "tie_word_embeddings": true
}

Tokenizer

Custom 48K vocabulary tokenizer with structured generation tokens built in:

{
  "backend": "tokenizers",
  "model_max_length": 2048,
  "bos_token": "<|bos|>",
  "eos_token": "<|eos|>",
  "pad_token": "<|pad|>",
  "unk_token": "<|unk|>",
  "extra_special_tokens": [
    "<|system|>", "<|user|>", "<|assistant|>",
    "<|think|>", "<|reasoning|>"
  ]
}

Usage

from transformers import AutoTokenizer
from MoA import MoAMetricLM, MoAMetricConfig

tokenizer = AutoTokenizer.from_pretrained("reaperdoesntknow/DiscoverLM-70M")
model = MoAMetricLM.from_pretrained("reaperdoesntknow/DiscoverLM-70M")

inputs = tokenizer("The triangle inequality guarantees that", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Chat Format

The tokenizer includes built-in special tokens for structured generation:

# Chat-style prompting
prompt = "<|system|>You are DiscoverLM, a small language model with metric attention.<|user|>What is the triangle inequality?<|assistant|><|think|><|reasoning|>"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)

Mathematical Foundations: Discrepancy Calculus (DISC)

The metric attention mechanism is grounded in Discrepancy Calculus — a measure-theoretic framework for singularity analysis developed by the author. The core operator:

$$Df(x) := \limsup_{r \downarrow 0} \sup_{0 < d(x,y) < r} \frac{|f(y) - f(x)|}{d(x,y)}$$

On metric-measure spaces (Axiom 11.1), this replaces the classical derivative with a scale-free local Lipschitz seminorm. The triangle inequality regularizer enforces that the learned attention geometry satisfies $d(a,c) \leq d(a,b) + d(b,c)$ across sampled triples — ensuring the distance function is a proper metric, not merely a similarity function.

The ball pruning mechanism (learnable per-head origins and radii) creates adaptive sparse attention patterns that emerge from gap geometry (Axiom 11.4): the gap set $\Delta(E) = {x : \theta^E(x) > \theta{E}(x)}$ defines where attention should be sparse. Tokens outside the ball are in the gap — structurally irrelevant to the current computation.

BlackHoleRoPE extends standard rotary position encoding with learned phase perturbations from a Fourier basis, maintaining the unitary property on Q/K while adding bounded amplitude modulation on V. The boundedness condition is the DISC discrepancy energy criterion: $E_{\text{disc}} < \infty$ ensures Lyapunov stability.

L2-star discrepancy for thermodynamic governance measures training dynamics as a discrepancy field — gradient magnitude distributions analyzed through the same measure-theoretic lens that DISC applies to function spaces.

Full theory: "On the Formal Analysis of Discrepancy Calculus" (CIx, 2026; Convergent Intelligence LLC: Research Division). Full methodology: Structure Over Scale (DOI: 10.57967/hf/8165).

Lineage

This architecture derives from research in metric-native neural computation:

• DISC — Discrepancy Calculus: measure-theoretic singularity analysis (CIx, 2025)
• MoA — Mixture-of-Attentions with triangle inequality enforcement
• BlackHoleRoPE — Learned rotary position encoding with bounded energy gating

Limitations

• Trained on 262K tokens — the architecture works, but this is a proof-of-concept scale. Generalization to unseen distributions is not yet validated.
• No eval split was used; training metrics only.
• 8 epochs over 64 batches means the model has seen each example multiple times. Overfitting is likely at this data scale.
• fp32 training only — bf16/fp16 behavior untested.

Citation

@misc{CILLC2026discoverLM,
  author = {Convergent Intelligence LLC: Research Division},
  title = {DiscoverLM-70M: Metric-Attention Mixture of Attentions with Triangle Inequality Enforcement},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/reaperdoesntknow/DiscoverLM-70M}
}

Author

Roy C — Convergent Intelligence LLC

HuggingFace: reaperdoesntknow

Convergent Intelligence Portfolio

Part of the Discover Series by Convergent Intelligence LLC: Research Division

Related Models

Top Models from Our Lab

Total Portfolio: 49 models, 22,598 total downloads

Last updated: 2026-03-28 12:55 UTC

From the Convergent Intelligence Portfolio

DistilQwen Collection — Our only BF16 series. Proof-weighted distillation from Qwen3-30B-A3B → 1.7B and 0.6B on H100. Three teacher variants (Instruct, Thinking, Coder), nine models, 2,788 combined downloads. The rest of the portfolio proves structure beats scale on CPU. This collection shows what happens when you give the methodology real hardware.

Top model: Qwen3-1.7B-Coder-Distilled-SFT — 508 downloads

Full methodology: Structure Over Scale (DOI: 10.57967/hf/8165)

Convergent Intelligence LLC: Research Division

Part of the reaperdoesntknow research portfolio — 49 models, 22,598 total downloads | Last refreshed: 2026-03-30 12:10 UTC