CMA 8M

Channel-Mixing Attention at 7.85 million parameters.

A compact generalist language model built to study channel mixing at small scale.

7.85M parameters | 1,024-token context | 4,096-token vocabulary | Safetensors

40,000 training steps | 20.97B trained tokens

Overview

CMA 8M is a 7,849,161-parameter Channel-Mixing Attention generalist causal language model trained for 40,000 steps. It pairs a compact nine-layer architecture with the unmodified native GPT-S 4,096-token tokenizer.

How CMA works

In each of the nine transformer blocks, causal token attention is followed by CMA in the feed-forward position. For every token, CMA reshapes the 288-channel hidden state into 12 chunks of 24, routes between those chunks with 3 heads, applies a signed blend and SiLU gate, then projects back to 288 channels.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "User01110/CMA-8M"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype="auto",
)

model.eval()

prompt = "Channel-mixing attention is"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Architecture

Input and output embeddings are tied.

Training mixture

All four sources are streamed natural-text corpora.

FineMath 4+ supplies high-quality mathematical explanations and reasoning as ordinary causal-language-model text.

Training trajectory

Across the logged validation checkpoints, normalized bits per byte (BPB) decreased from 3.7838 to 1.3520, while validation loss decreased from 8.3686 to 2.9903.

Evaluation

Zero-shot results at step 40,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with float32 weights and softmax. ArithMark uses float32 weights and its official raw continuation log-likelihood-sum rule.

WikiText-103 validation