Nexus-Erebus-135M
Nexus-Erebus-135M is a 134M-parameter language model trained from scratch with the Isabel Method (Ideoa Labs). It combines a digit-aware tokenizer with a compact Llama-style decoder to reach a balanced profile across commonsense reasoning, science question answering, and integer arithmetic at the 135M scale.
- Developed by: Malios Dark (Ideoa Labs)
- Model type: autoregressive decoder-only transformer (Llama-style)
- Language: English
- Parameters: 134M
Model architecture
- Decoder-only transformer with tied input/output embeddings, grouped-query attention (GQA), and rotary position embeddings (RoPE).
- Hidden size 640, 25 layers, 10 attention heads (5 key/value heads), context length 1024.
- Vocabulary of 32,000, byte-level BPE with digit isolation: every digit is its own token. This single choice is what makes small-scale arithmetic work, because the model reasons over individual digits instead of arbitrary multi-digit chunks.
Training data
Pretraining (from scratch, 2.66B tokens) on a 2025 English web mixture, by token share:
- Ultra-FineWeb, a re-filtered FineWeb successor (0.45)
- DCLM / DataComp-LM baseline web (0.30)
- Nemotron-CC, public sample (0.17)
- Synthetic integer arithmetic (0.05)
- Templated reasoning and word problems (0.03)
Fine-tuning corpus, built only from official training splits (no validation or test item is ever seen, so evaluation is contamination-free by construction):
- ARC-Easy, ARC-Challenge, OpenBookQA, SciQ, QASC, CommonsenseQA, HellaSwag, PIQA, WinoGrande.
- 150k synthetic arithmetic completions.
Training procedure
Pretraining used bf16 with a cosine schedule. Fine-tuning ran for 2 epochs at a low learning rate.
The important detail is the fine-tuning format. Every example is rendered in
completion form, that is context + correct continuation
(for example 8 + 5 = 13), rather than a letter-answer form such as
Question: ... Answer: D. This is deliberate: the benchmarks are scored by the
log-probability the model assigns to each candidate continuation, so the thing
worth optimizing is how likely the model makes the correct continuation. A model
trained to emit a letter learns to pick a letter, not to prefer the correct
completion, and lands near chance on the real scoring. Training in completion
form aligns the objective with the evaluation.
Evaluation
All benchmarks are run 0-shot and scored by log-likelihood multiple choice: each candidate answer is appended to the context and the highest-scoring continuation is selected.
| Benchmark | Metric | Score |
|---|---|---|
| HellaSwag | acc_norm | 30.61 |
| ARC-Easy | acc_norm | 48.61 |
| ARC-Challenge | acc_norm | 25.43 |
| PIQA | acc | 64.53 |
| ArithMark-2 | acc (length-normalized LL) | 66.68 |
Aggregate = ( HellaSwag + (ARC-Easy + ARC-Challenge) / 2 + PIQA + ArithMark-2 ) / 4 = 49.71
ArithMark-2 is the AxiomicLabs/ArithMark-2.0 held-out arithmetic set; it is a
synthetic benchmark and is never used for training.
A note on metric choice (for reproducibility)
HellaSwag and ARC are conventionally reported with length-normalized accuracy
(acc_norm), not raw accuracy (acc). Under raw accuracy, longer correct
answers are penalized simply for having more tokens, which understates these
tasks and can even push ARC-Challenge below random chance. We therefore report
acc_norm for HellaSwag and ARC, and acc for PIQA, following common practice.
Comparing an acc number for one model against an acc_norm number for another
produces non-comparable results; all figures above use the conventions above.
The Isabel Method
The Isabel Method is a small-model recipe with five parts:
- a digit-isolating tokenizer so arithmetic is learnable at small scale,
- from-scratch pretraining on recent, high-quality web corpora,
- arithmetic bootstrapping mixed into pretraining,
- contamination-free fine-tuning on official training splits only, in completion format, and
- evaluation that matches the scoring the benchmarks actually use.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-135M")
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-135M")
prompt = "8 + 5 ="
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=8)
print(tok.decode(out[0], skip_special_tokens=True))
Intended use
- Research on small language models, digit-aware tokenization for arithmetic, and from-scratch training recipes.
- English text generation and multiple-choice reasoning at the 135M scale.
Limitations
- English only, with a context length of 1024 tokens.
- As a compact research model, outputs should be verified before use in factual or downstream applications.
Creator: Malios Dark (Ideoa Labs).