Nexus-Erebus-3M
A 3-million-parameter language model trained from scratch. It is an arithmetic specialist: it handles integer arithmetic well for its size, and stays near chance on general tasks, at a footprint that runs almost anywhere.
How it works
The model reads and writes numbers least-significant-digit first, with a digit-atomic tokenizer.
Aligning carry propagation with reading order is what lets a model this small do arithmetic. The digit
reversal happens inside the tokenizer, so you pass and receive normal text. This requires
trust_remote_code=True.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-3M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-3M")
Results
Full test sets, 0-shot. acc_norm for multiple choice, accuracy for ArithMark-2.
| Task | Score |
|---|---|
| ArithMark-2 | 70.72 |
| PIQA | 51.41 |
| HellaSwag | 26.59 |
| ARC-Easy | 27.02 |
| ARC-Challenge | 20.65 |
The model is built for integer arithmetic. Non-arithmetic tasks sit near chance, which is expected at this size.
Reproduce the ArithMark-2 score
python benchmark_nexus_arithmark.py MaliosDark/Nexus-Erebus-3M
Training
Pretrained from scratch with a digit-atomic 4k tokenizer, least-significant-digit order, and a large stream of freshly generated synthetic integer arithmetic (addition, subtraction, multiplication, exact division, mixed and parenthesised multi-operator expressions), mixed with TinyStories for language. Then a short fine-tune on public benchmark train splits plus more arithmetic. No evaluation items were used at any stage. Fresh generation, never reusing a fixed pool, is what stops the model from memorising instead of learning.
Limitations
- It is an arithmetic specialist. On non-math tasks it performs near chance, by design.
- It reasons over integers in the ArithMark-2 style; it is not a general assistant.
License
MIT. Trained from scratch: 100% original weights, own tokenizer, no base model.