FEM-Inspired Causal Multi-Mesh LM

An experimental attention-free causal language model using:

• frozen injective Binary16 token coordinates;
• deterministic glyph/form observations;
• lossless short-span Structured Binary Tiles;
• a causal coarse history pyramid;
• latent scratch cells;
• explicit restriction and prolongation;
• shared iterative ConvGLU solvers.

This is a base model, not an instruction-tuned assistant.

Important interpretation

This release demonstrates that a heterogeneous causal multi-mesh system can learn coherent autoregressive text without attention.

It does not establish:

• literal finite-element equivalence for language;
• superiority over Transformers;
• better perplexity or throughput;
• autonomous reasoning;
• elimination of autoregression.

The implementation is best described as:

an FEM-inspired attention-free causal multi-mesh operator network.

Training checkpoint

The source directory name may contain 100b, but the authoritative training amount is the tokens_seen value above.

Architecture

The same text is represented as:

Binary16 fine token field
Structured Binary Tiles
glyph/form field
causal coarse hierarchy
latent scratch field

A completed coarse block may affect only later token positions.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "E6E831728/fem-multi-mesh-1p7b"

tokenizer = AutoTokenizer.from_pretrained(path)

model = AutoModelForCausalLM.from_pretrained(
    path,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer(
    "Gravity is",
    return_tensors="pt",
    add_special_tokens=False,
).to("cuda")

with torch.inference_mode():
    logits = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs.get(
            "attention_mask"
        ),
    ).logits

print(logits.shape)

Reference generation is available through:

generated = model.generate_simple(
    input_ids=inputs["input_ids"],
    max_new_tokens=9,
    temperature=0.0,
    eos_token_id=tokenizer.eos_token_id,
)

generated_text = tokenizer.decode(generated[0], skip_special_tokens=True)

print(generated_text)

Runtime limitations

• No attention KV cache is implemented.
• Generation recomputes the active context.
• Batched padded inference is not supported reliably; use batch size 1 for evaluation.
• The trained context is defined by config.block_size.
• The model's training loss API uses externally shifted labels. Standard evaluation should consume logits rather than rely on the training loss.
• The frozen form-codebook is stored in model.safetensors; it is not rebuilt during loading.

Intended use

Research on:

• attention-free language modeling;
• fixed token coordinates;
• heterogeneous text discretizations;
• explicit causal transfer operators;
• shared iterative computation.

Do not use for high-stakes decisions.

Audited standard language-model evaluation

This is evaluation of the standalone causal Multi-Mesh LM. No external document elements or VM calls are enabled. These scores do not establish superiority to a Transformer.

The model uses FEM-inspired multi-field operators; it does not implement or solve a physical finite-element PDE.