Learned Input Table Model Classic

Research checkpoint for the paper:

Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes

Model variant

This repository contains the learned input table baseline.

The model is a 32-layer decoder-only Transformer with:

• vocabulary size: 65,536
• model width: 1024
• number of layers: 32
• number of attention heads: 32
• context length: 1024
• rotary positional embeddings
• GELU activations
• untied trainable output projection

This baseline uses a standard trainable input embedding table of size:

65,536 x 1024 = 67,108,864 trainable input parameters

Intended use

This checkpoint is provided for reproducibility of the paper's controlled comparison. It is intended for research use only.

Loading example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Bochkov/llm-fix-min-baseline-learned-input-table-model-classic"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

prompt = "Question: What is the capital of United Kingdom?\nAnswer:"
input_ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)

with torch.no_grad():
    output_ids = model.generate(input_ids, max_new_tokens=3, do_sample=False)

print(tokenizer.decode(output_ids[0].tolist()))

Standardized base-model evaluation

The checkpoint was evaluated as a base causal language model with EleutherAI LM Evaluation Harness v0.4.10.

Evaluation protocol:

• Hugging Face backend: hf
• maximum context length: 1,024
• add_bos_token=False
• no chat template
• deterministic likelihood-based evaluation
• harness seeds: 0,1234,1234,1234
• base checkpoints only; no SFT or instruction checkpoints

The three paper checkpoints form the controlled architectural comparison. SmolLM2-135M and SmolLM2-360M are external reference models, not matched baselines: they use different architectures, tokenizers, training mixtures, and much larger pretraining budgets. SmolLM2-135M was trained on approximately 2T tokens and SmolLM2-360M on approximately 4T tokens, whereas the paper checkpoints saw approximately 16–17B tokens. Their scores therefore provide context for absolute capability and must not be interpreted as isolating the effect of the input parameterization.

Perplexity values should be interpreted especially cautiously across different tokenizers. The primary controlled comparison is among the three paper models, which share the same tokenizer, data pipeline, and architecture.

Input-interface audit

Unlike the two fixed-code variants, this control model uses a standard learned input embedding table. The table contains 67,108,864 trainable parameters and is expected to contain general real-valued entries rather than binary codes.

import torch
from transformers import AutoModelForCausalLM

repo_id = (
    "Bochkov/"
    "llm-fix-min-baseline-learned-input-table-model-classic"
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).cpu().eval()

embedding = model.get_input_embeddings()
weight = embedding.weight.detach()

print("shape:", tuple(weight.shape))
print("requires_grad:", embedding.weight.requires_grad)
print(
    "all entries binary:",
    bool(torch.all((weight == 0) | (weight == 1))),
)

assert tuple(weight.shape) == (65536, 1024)
assert embedding.weight.requires_grad is True
assert not torch.all((weight == 0) | (weight == 1))

Expected audit properties:

shape: (65536, 1024)
requires_grad: True
all entries binary: False

Limitations

This is a small research language model trained for architectural comparison. It is not instruction-tuned for safe deployment and should not be used as a production system.

Training data

The model was trained on the same FineWeb-Edu + Cosmopedia mixture used for the matched comparisons in the paper. Dataset terms and licenses are those of the original datasets.

🧑‍🔬 Citation & Concept

If you use this model or the underlying concepts in your research, please cite our work:

@misc{bochkov2026languagemodelstrainableinput,
      title={Language Models Without a Trainable Input Embedding Table: Learning from Fixed Minimal Binary Token Codes}, 
      author={A. Bochkov},
      year={2026},
      eprint={2605.09751},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.09751}, 
}