ConeML v16 6E · step 30,000
A base-pretraining research checkpoint accompanying Developmental Trajectories in a Sparse Language Model with 348M Active Parameters, Gerald Teeple · ConeML (2026).
Why this checkpoint is released
This artifact makes the reported developmental observation independently loadable. It is not an instruction-tuned assistant or a finished general-purpose release. The paper studies one lineage and a matched six-versus-nine-expert continuation; the results do not establish general model superiority or a causal curriculum advantage.
Evaluations
Frozen CPU-FP32 zero-shot screen: 1,048 examples with exactly the configurations and sample identities exported in the paper supplement. Scores below are subsets, not full-suite leaderboard results. SciQ includes its support passage.
| Task | Correct / examples | Accuracy |
|---|---|---|
| BLiMP | 377/536 | 70.34% |
| LAMBADA | 38/128 | 29.69% |
| PIQA | 77/128 | 60.16% |
| SciQ (with support) | 74/128 | 57.81% |
| ARC Easy | 58/128 | 45.31% |
Expanded LAMBADA: 1533/5025 (30.51%) on the additional partition and 1571/5153 (30.49%) on the full test. The additional partition excludes the repeatedly inspected original 128 examples. The report documents the pre-inference plan, paired uncertainty, selected-inventory overlaps and common-exclusion sensitivity. Inventory matches do not establish checkpoint-specific delivery.
Main Python execution remains 0/129 at this early checkpoint.
The matched 75k arms show mixed tradeoffs: 9E improves some public scores and parsing, while 6E retains lower held-out PPL and stronger narrative retention. At the later comparison, both measured Pythia references lead on the other four public tasks despite the LAMBADA advantage. Exact results and limitations are in the report and evaluation.json.
Intended format
Use raw text completion. No chat template is supplied because this checkpoint has not received instruction tuning. Do not wrap benchmark prompts as chat messages. The frozen evaluation used no added BOS token and no few-shot examples.
Loading
Validated with Python 3.12, PyTorch 2.12.1 and Transformers 5.12.1. This is a standard Mixtral model; no remote custom code is needed. The FP32 weights preserve the evaluated tensors. Lower precision or other software/backend versions may change outputs.
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
os.environ["HF_DEACTIVATE_ASYNC_LOAD"] = "1"
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
torch.set_num_threads(4)
repo = "ConeML/coneml-v16-6e-step30000"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo, dtype=torch.float32, device_map="cpu",
attn_implementation="sdpa", experts_implementation="eager",
).eval()
inputs = tokenizer("The old house stood at the end of the road.",
return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=128, do_sample=False,
pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The example's 128-token limit is a usage example, not the research scoring protocol. Exact benchmark settings and decoding are recorded in the supplement.
Architecture and exposure
- Mixtral-style decoder, 6 experts per layer, two selected per token.
- 820,233,216 stored parameters; approximately 348,374,016 active parameters per token. Active count is not dense-equivalent capacity or measured FLOPs.
- 30 layers, hidden width 1,024, 8 attention heads and 2 KV heads.
- Configured context: 16,384 tokens; vocabulary: 32,768.
- 3,932,160,000 nominal processed training positions. These are not unique tokens, and cross-model counts use different tokenizers.
- FP32 safetensors, tied embedding/output table stored once.
Strengths and known limitations
This checkpoint supports reproducible study of task-specific, non-monotonic development. Evaluation metrics can favor different checkpoints. One lineage, no independent training-seed replication, proprietary training construction, and incomplete contamination coverage limit generalization. Generated content can be wrong, repetitive or inappropriate. Low execution scores remain reported; parseability should not be treated as reliable coding ability. Context size is a configuration, not a claim of uniformly reliable long-context behavior.
Files, validation and license
Weights, inference configuration, tokenizer, this card, evaluation aggregates and file hashes are included. Optimizer state, loader/controller state, training data and private training configuration are excluded. The exported weights match the source tensors exactly; CPU-FP32 logits and short greedy generations matched on two conversion checks. These checks validate conversion, not broad capability.
Copyright © 2026 Gerald Teeple. Model weights and accompanying authored model-card material: CC BY-NC 4.0. See license. Third-party rights remain applicable. The separate paper supplement's evaluation wrapper has its own MIT license.
Report DOI: 10.5281/zenodo.23117644.