AB-EXT Binary16 — 1.711B parameters, 100B-token target

A base pretrained decoder-only causal language model released for research on trainable input embedding tables and fixed token identities.

It is not an instruction-tuned or preference-optimized assistant.

Research question

Can a shared contextual network learn useful language-modeling behavior without independently trainable token-specific input vectors?

The controlled family contains a learned-input model, a canonical 16-bit-code model, and an invertibly recoded GF2 model. They share the contextual backbone and output-head architecture, but not the same total trainable parameter count.

The results support viability, not performance equivalence. The fixed-code models retain substantial capability while the learned model performs better on several informative evaluations.

Model specification

The stored tensor-value count includes buffers and must not be reported as the trainable parameter count.

Input representation

Each token ID is represented by its canonical little-endian binary code:

$$ c(t)_j = \left\lfloor \frac{t}{2^j} \right\rfloor \bmod 2, \qquad j=0,\ldots,15. $$

Since the vocabulary contains 49,152 entries, an injective fixed-length binary code requires 16 bits.

The code is repeated 128 times to width 2048:

$$ x(t)= \underbrace{c(t)\Vert\cdots\Vert c(t)}_{128\text{ copies}}. $$

There are zero trainable input-interface parameters and no additional trainable input projection before the standard backbone. The backbone and the untied output vocabulary projection remain trainable.

The evaluated implementation stores the 49,152-by-16 codebook as a persistent non-trainable buffer. It is not literally lookup-free. “Minimal” describes fixed-length binary identity width, not the storage of the complete model or an entropy-optimal token code.

Training

• Training tokens: Approximately 100B according to the run report; target budget 100,000,000,000 prediction targets.
• Training precision: FP32 parameters with BF16 autocast in the supplied trainer.
• Reported recipe: AdamW; peak learning rate 0.00015; minimum scheduled learning rate 0.00001; 2000 warmup steps; cosine decay; weight decay 0.01; betas 0.9 and 0.95; gradient clipping 1.0.
• Reported launch geometry: two GPUs per run, microbatch eight per GPU, eight accumulation steps, sequence length 2048.

The launch geometry corresponds to 262,144 prediction targets per optimizer step. Exact final counts must come from the checkpoint, not from the requested budget.

The supplied sampler selects within-document windows from eligible documents of at least 2049 tokens. Sampling can repeat or overlap windows; 100B processed targets does not imply 100B unique corpus tokens. The original trainer does not fully restore per-rank sampling state on resume. A shared recipe alone does not establish identical realized sample order across interrupted runs.

The model weights were NOT initialized from SmolLM2. SmolLM2 supplies tokenizer artifacts, not pretrained model weights.

Evaluation results

These scores are transcribed from the supplied completed evaluation summary; the generator does not rerun benchmarks. Raw, unrounded harness outputs remain authoritative.

Accuracy entries are percentages. Their reported ± values are evaluation standard errors, not variation across training seeds. Perplexities and bits per byte are not percentages.

Audit and coverage limitations

The supplied audit reports identical sample/prompt multisets across all six models in each completed task group.

For MMLU 5-shot, 1,508 / 56,168 candidate log-likelihood requests were marked as truncated for each model, approximately 2.68%. These are candidate requests, not necessarily distinct questions. The displayed MMLU 5-shot score therefore includes truncated prompts.

No truncations were reported for the other groups by that audit. For WikiText rolling likelihood, this does not mean that whole documents fit into one model context: rolling windowing is part of scoring.

How to interpret SmolLM2 comparisons

All scores above are from the supplied local evaluation summary, not copied leaderboard scores.

The SmolLM2 technical report gives approximate training budgets of:

Source: https://arxiv.org/abs/2502.02737

These models differ in architecture, size, data, training schedule, and compute. They are quality references, not matched controls and not proof of a sample-efficiency advantage.

Usage

Review the custom Python files before enabling trust_remote_code=True. Use a tested Transformers version and pin the Hub revision for reproducible deployment.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = 'E6E831728/ab_ext_binary16'
# For published Hub use, pin revision to a reviewed commit.
revision = None

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    revision=revision,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    revision=revision,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer(
    "Gravity is",
    return_tensors="pt",
    add_special_tokens=False,
    return_attention_mask=True,
).to("cuda")

pad_id = tokenizer.pad_token_id
if pad_id is None:
    pad_id = tokenizer.eos_token_id

with torch.inference_mode():
    output = model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=32,
        do_sample=False,
        use_cache=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=pad_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

The implementation does not provide a KV cache. The trained context is 2048 tokens; the supplied generation adapter uses a sliding window when the context grows beyond its limit. This is not evidence of trained long-context capability.

Loss API

The original training model consumes already-shifted targets. The HF runtime is intended to expose the usual causal-LM convention with an internal label shift. Do not pass already-shifted labels to such a runtime.

Before fine-tuning, verify the actual runtime's loss implementation. Forward-logit equivalence does not by itself test label conventions.

Verification and integrity

The supplied verification logs report:

• successful BF16 loading and generation for all three releases;
• exactly matching original/exported forward logits on four short prompts for each model, in the tested verification configuration.

These are smoke and implementation-parity checks, not an exhaustive test across padding, context lengths, dtypes, or generation modes.

• Weight file: model.safetensors
• Weight SHA-256: c3e78f24f03dff36b6174985dc5189e8ce37a52685d022ade36fa68c3239801b
• Stored tensor values: 1,712,162,816
• Stored values by dtype: {"F32": 1712162816}
• Trainable parameter count: 1,711,376,384
• Persistent input-buffer values: 786,432

This card update does not modify the weights, tokenizer, model code, or configuration.

Limitations and intended use

• Research use and text completion; not a validated high-stakes assistant.

• One evaluated training run per input interface at this scale.

• Fixed-code and learned-input models are backbone-matched, not total-parameter-matched.

• No measured runtime or energy advantage is established by parameter counts alone.

• One GF2 recoding does not establish invariance to arbitrary codes.

• The output vocabulary matrix remains trainable and token-specific.

• Input-code structure is not fitted to the pretraining objective, but the tokenizer and its ID assignment can contain corpus-derived structure.

• Benchmark contamination has not been independently certified absent.

• Generated text can be false, biased, or harmful.

• Zero/one coding maps token ID zero to a zero input vector; the zero-offset GF2 transform preserves it. In the supplied bias-free architecture, a context made entirely of zero-code tokens gives uniform logits. This does not apply to arbitrary contexts ending in that token.

Attribution and licensing

Tokenizer artifacts are sourced from HuggingFaceTB/SmolLM2-1.7B.