Fixed Minimal Binary Code Model

This is an anonymized research checkpoint for the paper:

Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

Model variant

This repository contains the fixed minimal binary token-code model.

Instead of a trainable input embedding table, each token ID is represented by its exact minimal binary code.

For vocabulary size:

V = 65,536

the minimal injective binary code width is:

K = ceil(log2(V)) = 16

The 16-dimensional binary code is tiled to model width 1024.

The model therefore uses:

0 trainable input-embedding parameters

The output projection remains standard and trainable.

Architecture

• decoder-only Transformer
• vocabulary size: 65,536
• model width: 1024
• number of layers: 32
• number of attention heads: 32
• context length: 1024
• rotary positional embeddings
• GELU activations
• untied trainable output projection

Loading example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "E6E831728/fixed-minimal-binary-code"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

prompt = "Question: What is the capital of France?\nAnswer:"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False)

print(tokenizer.decode(output_ids[0].tolist()))

Standardized base-model evaluation

The checkpoint was evaluated as a base causal language model with EleutherAI LM Evaluation Harness v0.4.10.

Evaluation protocol:

• Hugging Face backend: hf
• maximum context length: 1,024
• add_bos_token=False
• no chat template
• deterministic likelihood-based evaluation
• harness seeds: 0,1234,1234,1234
• base checkpoints only; no SFT or instruction checkpoints

The three paper checkpoints form the controlled architectural comparison. SmolLM2-135M and SmolLM2-360M are external reference models, not matched baselines: they use different architectures, tokenizers, training mixtures, and much larger pretraining budgets. SmolLM2-135M was trained on approximately 2T tokens and SmolLM2-360M on approximately 4T tokens, whereas the paper checkpoints saw approximately 16–17B tokens. Their scores therefore provide context for absolute capability and must not be interpreted as isolating the effect of the input parameterization.

Perplexity values should be interpreted especially cautiously across different tokenizers. The primary controlled comparison is among the three paper models, which share the same tokenizer, data pipeline, and architecture.

Input-interface audit

This checkpoint stores a deterministic 65,536 × 16 binary codebook as a frozen nn.Embedding for computational convenience. During training, the table was initialized from the fixed token codes, marked with requires_grad=False, and excluded from the optimizer. It therefore contained 1,048,576 stored but non-trainable values and contributed zero trainable input parameters.

The released checkpoint can be audited directly:

import torch
from transformers import AutoModelForCausalLM

repo_id = "E6E831728/fixed-minimal-binary-code"

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).cpu().eval()

embedding = model.get_input_embeddings()
weight = embedding.weight.detach()

vocab_size, code_bits = weight.shape
ids = torch.arange(vocab_size, dtype=torch.long)
positions = torch.arange(code_bits, dtype=torch.long)

expected = ((ids[:, None] >> positions[None, :]) & 1).float()
expected[model.config.pad_token_id].zero_()

print("shape:", tuple(weight.shape))
print("unique values:", torch.unique(weight).tolist())
print("all entries binary:", bool(torch.all((weight == 0) | (weight == 1))))
print("exact canonical-code match:", bool(torch.equal(weight, expected)))
print("mismatching entries:", int((weight != expected).sum().item()))

assert tuple(weight.shape) == (65536, 16)
assert torch.all((weight == 0) | (weight == 1))
assert torch.equal(weight, expected)

Expected audit properties:

shape: (65536, 16)
unique values: [0.0, 1.0]
all entries binary: True
exact canonical-code match: True
mismatching entries: 0

Intended use

This checkpoint is provided for anonymous review and reproducibility of the paper's main claim: a trainable input embedding table is not necessary for useful language modeling in the studied regime.

Limitations

This model is a research checkpoint. It is not intended for deployment. It may produce incorrect, biased, unsafe, or nonsensical outputs.

Training data

The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.