mini-beatrix-1 — pretrain annealment point (pre-classroom)
The locked pretrain+anneal state of mini-beatrix-1, a 112.5M-parameter byte-level AlephLM: step 58,664, 17.301B bytes seen (0.3B wikitext warmup · 15B fineweb-edu · 2B anneal mix), fineweb-holdout val 1.045 bits/byte. This checkpoint is the fixed departure point for the staged "early-life curriculum" — later classroom checkpoints live in the training repo.
No tokenizer: she reads raw UTF-8 bytes (input_ids = byte values
0–255). Each position composes a byte trigram (dedicated pad row), so
"tokens" are learned inside the network. Sixteen pre-norm layers where
routing uses signed geometric addresses — sinh/Σcosh dispatch over
learned unit anchors, inhibition as a first-class citizen, no softmax-
over-choices, no top-k, no balance losses. Each layer carries an
anchored FFN bank born contributing exactly zero; layers 4/9/14 use a
linear-cost address read (CausalSplatHUB) instead of softmax attention.
Both elected themselves into load-bearing work: at this checkpoint,
removing the banks costs +2.25 bpb, removing the hub attention
+3.73 bpb (toggle ledger, fineweb holdout). The dual head's aleph
read is present with its gate folded to 1.0 (a verified semantic no-op,
max|logit diff| 2.4e-07) and contributes 0.0000 bpb here — it is the
live subject of the head-election experiment in the classroom phase.
Use
import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
"AbstractPhil/mini-beatrix-1", trust_remote_code=True).eval()
ids = torch.tensor([list("The history of mathematics begins".encode())])
out = m.generate(ids, max_new_tokens=96, do_sample=True,
temperature=0.7, top_p=0.95)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))
Bits-per-byte on your own text: pass labels=input_ids (HF shift
semantics are internal) and divide the returned loss (nats/byte) by
ln 2. No KV cache in this wrapper — generation recomputes the prefix
each step; for cached decode use the native stack below.
Honest notes
- The 2B anneal mix included dialogue in her chat template and a small
identity texture, so the bare model chats and knows her name —
behavior we have since ruled OUT of core corpora (conditioning belongs
in detachable arms; see the amoe-lora
arm system and
mini-beatrix-1/arms/in the training repo). - Small and early: conversational in shape, thin on knowledge,
confidently wrong at times. Curriculum probe baselines (P0–P8),
toggle ledgers, and lexicon-census reports for this exact checkpoint
are in the training repo under
mini-beatrix-1/reports/.
Code: github.com/AbstractEyes/alephllm · talk to her: alephllm-chat
Sibling
The next rung is complete: mini-beatrix-2s — 237M, full-splat (a governed constellation hub in every block), 16.101B tokens, finished 2026-08-31.
Licence
The weights are non-commercial: CC BY-NC-SA 4.0, as the whole Beatrix line is released; this generation trained on WikiText-103 (CC BY-SA) and FineWeb-Edu (ODC-By). The code files, the model design and the full training process are MIT (AbstractEyes/alephllm); relay.py is adapted from amoe-lora under Apache-2.0 (NOTICE). The full terms and the list of training sources: LICENSE.md.