DiscoverLM-70M-Base

A 70M parameter causal language model built on the Mixture-of-Attentions (MoA) architecture — distance-based metric attention that respects the triangle inequality by construction, not approximation.

Every attention head operates in a proper metric space. The geometry is enforced, not hoped for.

What Makes This Different

Standard transformers compute attention as a dot product: Q·Kᵀ. This has no geometric meaning — it's a bilinear form, not a distance. Two tokens can be "close" by dot product while violating basic metric properties.

MoA replaces this with negative squared distance under a learned diagonal Mahalanobis metric, then enforces the triangle inequality through a regularizer over random triples sampled during training. The result: attention weights reflect actual geometric proximity in a space where d(a,c) ≤ d(a,b) + d(b,c) holds.

This isn't a constraint that fights the model. It's structure the model uses.

Architecture

Input → Token Embedding (48K vocab, custom tokenizer)
  │
  ▼
┌──────────────────────────────────────────────────┐
│              MoA Block × 4                       │
│                                                  │
│  ┌─────────┐ ┌──────────┐ ┌────────┐ ┌────────┐ │
│  │  Local   │ │  Global  │ │Channel │ │  MQA   │ │
│  │  Conv    │ │  Metric  │ │  Mix   │ │ Metric │ │
│  │         │ │ (64 heads)│ │        │ │(64 Q)  │ │
│  └────┬────┘ └────┬─────┘ └───┬────┘ └───┬────┘ │
│       └──────┬────┴───────────┴───────────┘      │
│              ▼                                   │
│     Feature Gates + Token Router (top-2)         │
│              ▼                                   │
│        Residual + DropPath                       │
└──────────────────────┬───────────────────────────┘
                       ▼
         HyperFFN (SwiGLU + CausalConv + LowRank)
                       ▼
                   LayerNorm
                       ▼
┌──────────────────────────────────────────────────┐
│            MoA Language Model Head               │
│  (same 4-path mixture → SwiGLU → tied vocab)    │
└──────────────────────┬───────────────────────────┘
                       ▼
                 Logits (48,000)

Core Components

Metric Attention. Queries attend to keys via learned Mahalanobis distance. Each of 64 heads has an 8-dimensional head space with its own diagonal scaling, learnable ball origin, and adaptive radius for sparse pruning. Pairs outside the ball are masked before softmax.

Mixture-of-Attentions Routing. Four parallel paths per token — local depthwise convolution, full multi-head metric attention, gated channel mixing, and multi-query metric attention. A learned router selects top-2 paths per token position. Feature gates scale each path's output before mixing.

BlackHoleRoPE. Rotary position encoding with learned phase perturbations from a compact Fourier basis. Q/K rotations stay unitary. V amplitudes get bounded energy gating clamped to [0.5, 2.0] with optional discrepancy-state modulation.

HyperFFN. Three-branch feedforward: SwiGLU channel MLP, causal depthwise separable convolution, and gated low-rank bottleneck — routed per-token with top-2 sparse selection.

MoA LM Head. The vocabulary projection runs its own mixture-of-attentions (32 heads, head_dim=16) before projecting to logits through a SwiGLU transform. Weight-tied to the input embedding.

Parameter Budget

vs Standard Transformers

Training

Data

Usage

from transformers import AutoTokenizer
from MoA import MoAMetricLM, MoAMetricConfig

tokenizer = AutoTokenizer.from_pretrained("reaperdoesntknow/DiscoverLM-70M")
model = MoAMetricLM.from_pretrained("reaperdoesntknow/DiscoverLM-70M")

inputs = tokenizer("The triangle inequality guarantees that", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Chat Format

The tokenizer includes built-in special tokens for structured generation:

# Chat-style prompting
prompt = "<|system|>You are DiscoverLM, a small language model with metric attention.<|user|>What is the triangle inequality?<|assistant|><|think|><|reasoning|>"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)

Mathematical Foundation

The metric attention mechanism is grounded in the Discrepancy Calculus (DISC), a measure-theoretic framework for singularity analysis developed by the author. The triangle inequality regularizer enforces that the learned attention geometry satisfies d(a,c) ≤ d(a,b) + d(b,c) across sampled triples, ensuring the distance function used for attention scoring is a proper metric — not merely a similarity function.

The ball pruning mechanism (learnable per-head origins and radii) creates adaptive sparse attention patterns that emerge from the geometry itself rather than from fixed masking heuristics.

BlackHoleRoPE extends standard rotary position encoding with learned phase perturbations synthesized from a Fourier basis, maintaining the unitary property on Q/K while adding bounded amplitude modulation on V — ensuring position-dependent energy gating stays within Lyapunov-stable bounds.

Lineage

This architecture derives from research in metric-native neural computation:

• DISC — Discrepancy Calculus: measure-theoretic singularity analysis (CIx, 2025)
• MoA — Mixture-of-Attentions with triangle inequality enforcement
• BlackHoleRoPE — Learned rotary position encoding with bounded energy gating

Limitations

• Trained on 262K tokens — the architecture works, but this is a proof-of-concept scale. Generalization to unseen distributions is not yet validated.
• No eval split was used; training metrics only.
• 8 epochs over 64 batches means the model has seen each example multiple times. Overfitting is likely at this data scale.
• fp32 training only — bf16/fp16 behavior untested.

Citation

@misc{CILLC2026discoverLM,
  author = {Convergent Intelligence LLC: Research Division},
  title = {DiscoverLM-70M: Metric-Attention Mixture of Attentions with Triangle Inequality Enforcement},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/reaperdoesntknow/DiscoverLM-70M}
}

Author

Roy C — Convergent Intelligence LLC

HuggingFace: reaperdoesntknow

Convergent Intelligence Portfolio

Part of the Discover Series by Convergent Intelligence LLC: Research Division

Related Models

Top Models from Our Lab

Total Portfolio: 41 models | 2,781 total downloads

Last updated: 2026-03-28 12:56 UTC

From the Convergent Intelligence Portfolio

DistilQwen Collection — Our only BF16 series. Proof-weighted distillation from Qwen3-30B-A3B → 1.7B and 0.6B on H100. Three teacher variants (Instruct, Thinking, Coder), nine models, 2,788 combined downloads. The rest of the portfolio proves structure beats scale on CPU. This collection shows what happens when you give the methodology real hardware.

Top model: Qwen3-1.7B-Coder-Distilled-SFT — 508 downloads

Full methodology: Structure Over Scale (DOI: 10.57967/hf/8165)

Convergent Intelligence LLC: Research Division