Nero XS

A 2,996,480-parameter recurrent XSA language model trained from scratch with MLX on Apple Silicon.

Quick start (MLX)

pip install "mlx>=0.32" "transformers>=5" huggingface_hub numpy

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("j0no12/Nero-XS")
sys.path.insert(0, model_dir)

from modeling_nero_xs_mlx import load_model, generate

model, tokenizer = load_model(model_dir)
text = generate(
    model,
    tokenizer,
    prompt="The future of small language models is",
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    seed=7,
)
print(text)

Nero XS uses the custom MLX architecture class NeroXSAForCausalLM. It is not a drop-in Transformers AutoModelForCausalLM checkpoint. The complete standalone inference implementation is included in modeling_nero_xs_mlx.py.

Results

All scores are zero-shot normalized accuracy on the full evaluation split.

HellaSwag, ARC, and PIQA were evaluated with lm-eval 0.4.12 using exact continuation likelihood, zero few-shot examples, acc_norm, and the full task splits. ArithMark-3 used the official independent-tokenization continuation protocol and acc_norm. Raw result files are included under evals/.

Takeaway: Nero XS is meaningfully above random baselines on several tasks despite storing fewer than three million parameters. It remains an extremely small research model, and its benchmark differences should be interpreted with the reported sample sizes rather than as broad language competence.

Architecture

XSA removes from each attended head output its projection onto the current-token value vector. The four middle physical blocks are applied twice, providing fourteen effective block transformations without storing a second set of recurrent weights.

Training story

1. DCLM base stage

• Exact stage budget: 4,999,999,488 tokens.
• Sequence length 256, batch size 16, seed 7.
• AdamW: peak learning rate 0.012, 10M-token warmup, cosine decay to 0.0012, weight decay 0.1, betas (0.9, 0.95), epsilon 1e-8.
• Source-disjoint train/dev/test token streams produced from the local DCLM preparation pipeline.

2. FinePhrase continued pre-training

• Exact stage budget: 4,999,999,488 tokens.
• Balanced equally across FinePhrase faq, math, table, and tutorial configurations during corpus preparation.
• Used only rollout_results[0].text; the source text field was ignored.
• Peak learning rate 0.0012, no warmup, cosine decay to 0.00012.
• Preserved the base model and complete AdamW optimizer state; reset only the dataset sampler and stage token counter.

Total recorded training exposure across both stages is 9,999,998,976 tokens.

Data lineage

The FinePhrase preparation accepted 886,182 documents across the four configurations before assigning source-disjoint train/dev/test splits. Dataset sampling beyond each prepared stream's unique size means the training-token total is exposure, not unique-token count.

Evaluation details

The released checkpoint was chosen by the highest unweighted mean across the five reported public tasks among the already evaluated training checkpoints. This is a benchmark-informed release selection, so the table is descriptive rather than an untouched final estimate.

Reproducibility

model.npz and model.safetensors contain the same 103 FP32 tensors. The standalone release implementation was checked against the training implementation with exact logit equality (max_abs_diff = 0.0) on a fixed input.

Limitations

• Fewer than three million parameters sharply limits factual recall, instruction following, coherence, and long-form generation.
• Training used 256-token sequences. Longer contexts are accepted by the deterministic coordinate implementation but are not established training-distribution behavior.
• The 2K tokenizer favors parameter efficiency over tokenization efficiency and has limited dedicated multi-digit coverage.
• Generation uses full-prefix recomputation and has no KV cache.
• FinePhrase content is synthetic and can transfer teacher errors or style.
• No independent human evaluation, safety evaluation, or comprehensive benchmark-contamination audit is published.
• Public benchmarks influenced checkpoint selection; do not treat these five scores as an untouched test set.
• This base model is not instruction-tuned and should not be expected to answer prompts like a chat assistant.

Files

Citation

@misc{neroxs2026,
  author       = {j0no12},
  title        = {Nero XS: A 2.996M-Parameter Recurrent XSA Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-XS}},
  note         = {Trained from scratch with MLX on Apple Silicon}
}

License

The original Nero XS model weights and repository material are released under CC-BY-4.0. Attribution is required. The DCLM source is recorded as CC-BY-4.0, while FinePhrase is ODC-BY; those upstream dataset terms and attribution requirements remain applicable to their respective source material.