Nero XS 2

A 2,993,152-parameter recurrent grouped-query XSA language model trained from scratch with PyTorch/XLA on TPU v5e-8.

Quick start (PyTorch)

pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub

from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("j0no12/Nero-XS-2")
sys.path.insert(0, model_dir)

from modeling_nero_xs_2 import load_model, generate

model, tokenizer = load_model(model_dir, device="cpu")
text = generate(
    model,
    tokenizer,
    prompt="The future of small language models is",
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    seed=7,
)
print(text)

Nero XS 2 uses the custom PyTorch class NeroXSAForCausalLM. It is not a drop-in Transformers AutoModelForCausalLM checkpoint. The complete standalone implementation is included in modeling_nero_xs_2.py.

Results

All scores are zero-shot normalized accuracy on the full evaluation split.

HellaSwag, ARC, and PIQA were evaluated with lm-eval 0.4.12 using exact continuation likelihood, zero few-shot examples, acc_norm, and full task splits. ArithMark-3 used the official independent-tokenization continuation protocol and acc_norm. Raw results are included under evals/.

Takeaway: this recovery checkpoint improves four of five public tasks over the original Nero-XS-2 checkpoint and raises the unweighted mean 0.25 percentage points above Nero XS. It is not established as a top-three or state-of-the-art model. Frozen-dev NLL worsened during recovery even as public multiple-choice scores improved, so these gains should be interpreted cautiously.

Architecture

The model shares four physical middle blocks across two recurrent passes. Each pass has a learned embedding and channel-wise gate. XSA subtracts the component of each attended head output aligned with the current-token value vector.

Training story

1. Nero-XS-2 pretraining

• Exact target: 25,000,148,992 tokens.
• Sequence length 256; global batch 2,048 sequences, or 524,288 tokens per optimizer step.
• AdamW state, scheduler position, sampler position, and RNG state were preserved across TPU sessions.
• The original 2,994,179-parameter graph included EngramLite hashed bigram/trigram memory and FFN width 531.
• Original frozen-dev NLL: 2.4375.

2. Engram-anneal recovery

• Preserved 84 compatible tensors from the 25B checkpoint.
• Expanded 30 FFN tensors from width 531 to 582, copying every trained channel and randomly initializing only 51 new channels.
• Froze and annealed EngramLite to zero over 1,500,037,632 tokens.
• Continued the final Engram-free graph through 2,500,329,472 recovery tokens.
• Total recorded exposure: 27,500,478,464 tokens.
• Recovery throughput: 493,879 tokens/s on TPU v5e-8.
• Final frozen-dev NLL: 2.65625, worse than before recovery.

The released checkpoint is the final recovery checkpoint. Public tasks were evaluated only after the recovery design and checkpoint were fixed.

Data lineage

The immutable prepared corpus contained 495,134,047 unique training tokens and was sampled repeatedly to the training budget.

Preparation rejected any document sharing a normalized 13-word sequence with HellaSwag, ARC, PIQA, or ArithMark-3 prompts or answers. The prepared training stream SHA-256 is cd3ac872bf365c364f82348703635613fb78b4aa927b7e9f0d7db1a89c729ebf.

Evaluation details

Reproducibility

The released safetensors file was produced directly from the portable final checkpoint. The standalone implementation was checked against the training implementation on a fixed input before release.

Limitations

• Fewer than three million parameters sharply limits factual recall, instruction following, coherence, and long-form generation.
• Training used 256-token sequences. Longer contexts are outside the established training distribution.
• The 2K tokenizer favors parameter efficiency over tokenization efficiency.
• Generation uses full-prefix recomputation and has no KV cache.
• This is a base model, not an instruction-tuned chat assistant.
• Recovery worsened frozen-dev NLL from 2.4375 to 2.65625 despite improving four public tasks.
• No independent human evaluation or comprehensive safety evaluation is published.
• The scores are not independently verified leaderboard submissions.

Files

Citation

@misc{neroxs2_2026,
  author       = {j0no12},
  title        = {Nero XS 2: A 2.993M-Parameter Recurrent Grouped-Query XSA Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-XS-2}},
  note         = {Trained from scratch with PyTorch/XLA on TPU v5e-8}
}

License

The original Nero XS 2 weights and repository material are released under CC-BY-4.0. Attribution is required. Upstream dataset terms and attribution requirements remain applicable to their respective source material.