CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1

CascaMini-350M-C24 is a 339,788,800-parameter base language model trained from scratch for 15,000,010,752 tokens.

This is the first production CascaMini-350M release. It is a pretrained base model, not an instruction-tuned or chat-aligned assistant.

Model summary

The Hugging Face repository is a parity-checked conversion of the accepted custom CascaMini release to standard Transformers LlamaForCausalLM format.

Tokenizer

The model uses the SmolLM2-49K / cosmo2-tokenizer vocabulary.

The original tokenizer vocabulary and merges are retained. The Hugging Face model_max_length metadata is set to 2,048 to match the model's actual trained context length.

Training

Data mixture — CascaMini-CoreMix-v1 / M3

The CoreMix source documents were filtered and materialised before training. The final training sampler used the frozen M3 source weights above.

See TRAINING_DATA_AND_ATTRIBUTION.md for the retained source revisions, licence/provenance audit, and third-party data notes.

Optimizer

The production optimizer was VO Muon: Muon applied only to the attention value and output projection matrices, with auxiliary AdamW for the remaining parameters.

Muon:

• peak LR: 0.0025
• minimum LR: 0.00025
• cosine schedule
• warmup: 0
• momentum: 0.95
• Newton-Schulz iterations: 5
• Muon weight decay: 0

Auxiliary AdamW:

• peak LR: 2e-4
• minimum LR: 2e-5
• warmup: 100 steps
• weight decay: 0.1
• betas: (0.9, 0.95)
• eps: 1e-8

Training used 8 GPUs, BF16, context length 2,048, and 65,536 tokens per global update.

Reported training throughput was approximately 512,637 tokens/s and the trainer-reported elapsed time was approximately 9.31 hours.

Final evaluation

The final checkpoint was also the best fixed-validation checkpoint.

Per-source BPB

3B -> 15B scaling

Using the same production-selected VO-Muon recipe:

Repetition stress tests

Long continuation repetition remains the clearest known weakness.

These are deliberately long-generation stress diagnostics, not downstream task scores.

No standard downstream benchmark suite is claimed in this release yet.

Geometry diagnostics

Final diagnostic values:

• L24 residual RMS: 14.9823
• logit RMS: 6.4040
• mean Q stable rank: 18.28
• mean K stable rank: 13.75
• mean V stable rank: 76.87
• mean O stable rank: 123.80

The VO-Muon training line remained numerically stable through the full 15B schedule.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "beardymcgee/CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    device_map="auto",
)

prompt = "The most important property of a reproducible scientific experiment is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=96,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.05,
)

print(tokenizer.decode(output[0], skip_special_tokens=False))

For raw greedy decoding:

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
)

Intended use

This release is intended for:

• language-model research;
• evaluation and reproducibility work;
• fine-tuning / adaptation experiments;
• studying small-model data, optimizer, and scaling behaviour.

It is not presented as a factual authority, safety-aligned assistant, or production decision system.

Limitations

This is a small base model. It can:

• hallucinate or state false information confidently;
• repeat phrases or enter repetitive continuation patterns;
• generate biased, offensive, unsafe, or otherwise undesirable text;
• reproduce fragments resembling training material;
• perform poorly on tasks requiring knowledge or reasoning beyond its scale;
• fail to follow instructions because it was not instruction-tuned.

The long-generation repetition measurements above are published explicitly because repetition remains a known weakness of this model family.

Training-data provenance and licensing

The canonical CoreMix JSONLs retained licence_evidence_mode and licence_fields for 100% of the 6,699,937 CoreMix training records audited for this release.

The corpus nevertheless contains multiple third-party licence classes, including CC BY, CC BY-SA, public-domain/CC0/GFDL material, and source code under many permissive SPDX licences. FineWeb-Edu is tracked separately through its upstream dataset terms.

For that reason this model card uses:

license: other

rather than implying that one simple permissive licence accurately describes all provenance and downstream rights questions associated with the release.

See MODEL_RELEASE_NOTICE.md and TRAINING_DATA_AND_ATTRIBUTION.md.

No training dataset is redistributed in this model repository.

Release integrity

Original accepted release identity:

CascaMini-350M-C24-CoreMix-v1-15B-VO-Muon-v1

Original accepted model_state.pt SHA-256:

4e40cca125c8fb4b866632708af5e0d5862cf66413bbbeb0a31c6a0385b30d65

The original release passed:

• strict state-dict loading;
• exact 339,788,800 parameter count;
• tied embedding/head verification;
• local tokenizer loading;
• deterministic independent cold loads;
• logit fingerprint matching.

The Hugging Face conversion additionally passed original-vs-standard-Llama logit parity and a fresh local AutoModelForCausalLM.from_pretrained() load.

Project status

This is the frozen 350M first-production release. Future CascaMini work may use larger or more diverse data corpora, but those are separate experimental lineages and do not change this release.