OpenSML-Base-175M

Author: William Zebrowski · 175.6M parameters · 17.30 billion pretraining token positions

Run locally · Technical report · Evaluation results · Checkpoint provenance

An English pretrained causal language model built from random initialization on educational web text, Wikipedia, synthetic educational writing and Python. This is the pretrained ancestor of OpenSML-JSONLite-175M-Instruct. It supports text completion and further fine-tuning. Its weights contain Stage A pretraining and continuation only, without QA/extraction SFT or a JSON adapter.

Download and complete text

The repository includes model.safetensors, frozen tokenizer, configuration, native architecture source and a standalone completion runner. The verified runtime is Apple Silicon macOS, Python 3.11 and stock MLX 0.32.3.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub
# Authenticate if access requires it: hf auth login
hf download OpenSML/OpenSML-Base-175M --local-dir ./OpenSML-Base-175M
cd OpenSML-Base-175M
python -m pip install -r requirements.txt
python inference.py --prompt "Plants need sunlight because" --max-tokens 64

The runner applies no chat template. It takes a raw prefix and prints its continuation. Use --prompt-file prompt.txt --json for a UTF-8 prefix and a machine-readable generation record. Greedy decoding is the default; optional sampling uses --temperature 0.7 --seed 42. Prompt plus generation budget must fit 2,048 tokens; overlong requests are rejected without silent truncation.

This custom architecture uses the supplied native runner, rather than transformers.AutoModel or mlx_lm.load. CUDA, CPU-only, quantized and converted exports have not been validated. After downloading artifacts and installing dependencies, generation runs offline. The root runner defaults to the adjacent weight file and verifies its SHA-256 and source bundle. The source bundle includes all native tokenizer integrity companions. See RUN_GUIDE.md.

Model specification

The earlier OpenSML tokenizer is reused unchanged. OpenSML-150M weights did not initialize this model. The native tokenizer's structural IDs are pad 0, document-end 1, turn-start 2 and turn-end 3; the base runner uses raw text.

Pretraining

The selected checkpoint processed 17,300,111,360 lifetime token positions. The intended sampling mixture was:

These are configured token-sampling weights, not unique-document shares. Source revisions, score thresholds, filters and split policy are in corpus.json and the report. Training uses next-token prediction on a five-Mac Apple Silicon cluster with Thunderbolt/RDMA. AdamW uses betas (0.9, 0.95), epsilon 1e-8, weight decay 0.1 and gradient clipping 1.0, with FP32 master weights and moments.

Nominal peak LR was 3e-4; an adaptive plateau controller caps the scheduled rate. The effective cap at the selected checkpoint is 3.221225472e-5, after nine controller reductions. Validation occurs at 50M-token boundaries, with patience 12 evaluations, cooldown 10 and minimum delta 0.002. The original budget and continuation target were higher than the selected artifact's 17.30B positions; this card does not claim a completed 30B run. See the exact frozen optimizer and scheduler record in checkpoint_metadata.json.

Training run

The 175.6M-parameter base was trained from random initialization using next-token prediction across the educational web, Wikipedia, Cosmopedia and Python mixture. Five Apple Silicon Macs carried out distributed pretraining over Thunderbolt/RDMA. Validation was measured every 50M tokens. The curves show a steep early loss reduction followed by slower improvement; the published base was selected using held-out validation.

Training loss and held-out validation during pretraining. Left: raw training loss and exponential moving average. Right: held-out validation loss.

Recorded evaluation

These are historical document-prediction measurements for this exact artifact. The fixed panel has 32 batches per source, batch size 2, five sources and 2,048-token sequences; aggregate loss follows the training mixture. The exact validation fingerprint and source losses are in EVALUATION_RESULTS.json.

No verified downstream benchmark result for this exact weight hash was found in the retained benchmark-results inventory. JSONLite extraction scores and other OpenSML checkpoint ARC/IFEval scores are not attributed to this base. Historical qualitative samples are in PRETRAINING_SAMPLES.json; they show repetition and unsupported continuations as well as plausible first completions. They are not factual accuracy or instruction-following benchmarks.

Intended use and limitations

Use as a research text-completion model or a starting point for adaptation. Pretraining does not establish reliable instruction following, conversation, structured extraction, code execution correctness or calibrated factuality. Outputs can repeat, invent facts and continue unrelated document patterns. The small context and parameter budget also limit long-document work. Safety, privacy, multilingual transfer and prompt-injection robustness have not been comprehensively evaluated.

Identity and reproducibility

Published model.safetensors has 351,242,342 bytes and SHA-256:

5151567e266244c9e32fd6e01775192633abe227320269ecdefe28ce43e9ccc3

The repository name identifies the model; exact training counters and metadata are kept in provenance. The tokenizer fingerprint is 1194292c6d906ac19e6f01c6ad2b42825143fad426a22aa8856d903776701ecb. SHA256SUMS checks downloadable artifacts. The inference source bundle contains model, kernels and tokenizer assets, without SFT data, optimizer states or cluster credentials. Package smoke tests establish runnable native completion, not a reproduction of the historical validation panel or a new blind benchmark.

Licensing and attribution

Author-controlled source contributions use the included Apache-2.0 license. The model-weight license remains unassigned; the source license does not grant additional weight rights. YAML therefore omits a model-weight license identifier. Training sources retain their own terms. See THIRD_PARTY_NOTICES.md, corpus.json, the report and CITATION.cff for notices. Dataset declarations do not establish blanket underlying-content redistribution clearance.

Native JSON descendant update

The current OpenSML-JSONLite-175M-Instruct adds a joint full-weight JSON extraction SFT to its earlier grounded-QA, extraction and native JSON ancestry. The latest stage mixes original source/schema/JSON TRAIN replay, multiple-value examples and authored prose. The released main checkpoint inherits 19,200 presentations from a 21,144-presentation experiment. It directly generates the complete JSON object; validation follows without assembly or repair.

On the same previously observed 1,444-request SNIPS-derived evaluation, the descendant achieves 1,271/1,444 (88.02%) complete correctness, 94.17% field accuracy, 92.88% extracted-value F1, 92.42% precision, 93.35% recall, 95.22% missing-field accuracy and 100% schema validity. The observed 60-case playground scores 49/60. Compared with the previous release, 96 requests improve and 24 regress. These are descendant results on repeatedly observed research panels, not scores of this base or a fresh blind benchmark. See the current JSONLite technical report for the full provenance, protocol and limitations.

The pretrained base weights, tokenizer, inference runner and source bundle are unchanged.