open-1b-sft (MLX, bf16)
open-1b is a 1.6 billion parameter decoder-only language model pretrained on 400B tokens. It is the first language model whose training can be independently verified: it is released with its complete pretraining dataset, training and evaluation code, intermediate checkpoints at 100-step intervals, and a canonical state hash for every one of the 80,957 optimizer steps that produced it.
This repository is an MLX conversion of
Gensyn/open-1b-sft — the
supervised fine-tune (SFT) of the family — for Apple Silicon, runnable with
mlx-lm and
LM Studio with no custom code. The verifiable artefact
is the reference checkpoint;
this conversion is for convenient local inference and is not itself covered by
the published state hashes.
The open-1b family
| Model | What it is |
|---|---|
Gensyn/open-1b-base | Pretraining checkpoint at step 80,957, 400B tokens |
Gensyn/open-1b-midtrained-93B | Midtrained checkpoint: continued pretraining of the base model on 93B additional tokens |
Gensyn/open-1b-sft | Supervised fine-tune on allenai/tulu-3-sft-olmo-2-mixture-0225, the chat-capable model |
Gensyn/open-1b-sft-mlx-bf16 | This repo: open-1b-sft converted to MLX (bf16) |
Model sources
- Verification record: https://open1b.gensyn.ai
- Tech report: open1b.gensyn.ai/open1b-tech-report.pdf
- Training code, configs and RepOps: https://github.com/gensyn-ai/open-transformers
- Audit tool: https://github.com/gensyn-ai/pretraining-audit-cli
How to use
With mlx-lm:
pip install mlx-lm
python -m mlx_lm.chat --model Gensyn/open-1b-sft-mlx-bf16
from mlx_lm import load, generate
model, tokenizer = load("Gensyn/open-1b-sft-mlx-bf16")
messages = [{"role": "user", "content": "Explain verifiable training in one paragraph."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt, max_tokens=256))
In LM Studio, search for Gensyn/open-1b-sft-mlx-bf16, download it and load it.
LM Studio samples at temperature > 0 by default and ignores
generation_config.json; we recommend setting Temperature to 0 in the
model's settings (see Limitations). mlx-lm already decodes greedily by
default.
Conversion details
Weights. These are the bf16 master weights of open-1b-sft. The model
was trained with int8 W8A8 quantization-aware training (LSQ); the reference
transformers port emulates that int8 grid by default, but native MLX has no
equivalent, so this conversion drops the learned weight_scale tensors and
runs plain bf16 GEMMs — equivalent to the reference with
quantized_forward=False. Greedy outputs match that reference token for
token on our probes. Expect small differences from the reference's default
int8-emulated forward.
Architecture as a built-in mlx-lm type. mlx-lm has no open1b
architecture, and apps like LM Studio refuse checkpoints that need custom
model code. The weights are therefore re-expressed as mlx-lm's built-in
step3p5 architecture — which is why config.json says
"model_type": "step3p5". This is still open-1b, not a Step model. Every
rewrite is exact:
| open-1b feature | expressed as |
|---|---|
| embedding RMSNorm | baked into the embedding table (it depends only on the token) |
| interleaved-pair RoPE | q_proj/k_proj rows permuted within each head → half-split RoPE (exact, because the QK-norm before RoPE has no gain) |
| gain-free per-head QK-norm | q_norm / k_norm with unit weights |
| fused gate/up projection | separate gate_proj / up_proj |
| dense SwiGLU MLP | moe_layers_enum: "-1" (no MoE layers), no attention gate |
The one approximation: open-1b's sliding-window layers (19 of 24) use a
window aligned to 32-token blocks (each query sees 481–512 keys), while
mlx-lm uses a plain 512-key window. Outputs are identical up to 481 tokens
of context and near-identical beyond it (per-token loss on a 1.5k-token
document unchanged to three decimals).
Tokenizer. Identical to the reference except for one removed BPE merge
("Ċ" + "Ċ" → "ĊĊ", id 408). A \n\n at the very end of an encoded string
(exactly where the chat template's generation prompt ends) otherwise becomes
a token the model never saw there, producing empty replies. The reference
repo fixes this with custom tokenizer code; removing the merge fixes it for
any standard tokenizer consumer. Text where \n\n is followed by content
tokenizes exactly as in training.
Conversion records: post-conversion-fixes.json (what changed here, with
hashes) and provenance/ (the upstream converter, its hash-pinned manifest
and metadata history).
Chat format
<|start_header|>{{role}}<|end_header|>
{{content}}<|eot|>
Special tokens <|start_header|> (id 2), <|end_header|> (id 3) and <|eot|>
(id 4, the eos). No BOS prefix. The generation prompt ends in
<|start_header|>assistant<|end_header|>\n\n, exactly as in training.
Limitations
This is an SFT-only model (no RLHF or preference tuning) at 1.6B parameters:
a helpful but lightly-tuned research assistant, not a production chat model.
It has a learned tendency to hedge on simple factual questions ("As of my
knowledge cutoff…", "your question is underspecified…"). The tendency is
stronger under sampling and on casually phrased prompts. For example,
What is the capital of France? gets a plain answer, while
what is the capital of france hedges even with greedy decoding. This is a
property of the trained model: the reference transformers checkpoint
behaves the same way. It is not caused by the conversion.
License
Apache 2.0 — see LICENSE.