ANKA-50M RMW3

A 48.94M-parameter English causal language model trained from random initialization on 5 billion tokens using a single NVIDIA A100 80 GB GPU.

ANKA-50M RMW3 explores how much language modeling capability can be obtained under a strict 50-million-parameter budget. It combines a parameter-shared recurrent Transformer backbone with two learned components—Reflect and Morph—and was trained with a mixture that includes WorldPairs relational examples.

Model type: pretrained causal base model. This checkpoint is not instruction-tuned, chat-aligned, or safety-aligned. Its native behavior is text continuation.

ANKA-50M RMW3 GITHUB

link: https://github.com/mamalito345/ANKA-50M-RMW3

Highlights

• 48,944,657 unique trainable parameters, including the tied embedding/output matrix.
• 5,000,000,000 training tokens from random initialization; no pretrained initialization, model merging, or distillation.
• One NVIDIA A100-SXM4 80 GB, not a multi-GPU run.
• 117,899 tokens/second average measured training throughput.
• 11 h 46 m 49 s total training wall time, equal to approximately 11.78 GPU-hours.
• Approximately 424.4 million training tokens per GPU-hour.
• Exact PyTorch architecture, tokenizer, SafeTensors checkpoint, evaluation reports, and architecture ablations are included.

Training efficiency

Training efficiency was a first-class goal of this release. The complete 5B-token base run used BF16 precision and an in-memory data pipeline on a single accelerator.

The throughput figure is the average recorded by the training loop for the full run. Corpus construction and tokenizer training are separate preprocessing stages and are not included in the 42,409-second model-training time.

Architecture

The 13 physical Transformer blocks are applied twice with shared parameters. This produces 26 effective block applications without duplicating the backbone weights.

Reflect

During the second recurrent pass, each Reflect unit compares the current hidden state with the corresponding state saved during the first pass. A gated rank-32 transformation turns that discrepancy into a learned correction.

Morph

Four sparse Morph stages are distributed across the effective depth. At each stage, a learned router assigns each token to one of four rank-368 experts. Only the selected expert is evaluated for that token, and its output is added through a learned scale.

WorldPairs

WorldPairs is a data-side component containing paired descriptions of controlled world-state transformations. It was included to expose the model to explicit relational changes rather than relying only on unpaired next-token examples.

Architecture ablations

The following results come from the included 4,096-token architecture audit. They are diagnostic ablations, not replacements for the official benchmarks.

Additional observations:

• Reflect improved token-level NLL for 70.70% of the audited tokens.
• All 13/13 Reflect units had a positive individual ablation contribution.
• Morph improved token-level NLL for 65.33% of the audited tokens.
• All 4/4 Morph stages had a positive individual ablation contribution.
• The trained operating scale of 1.0 was the best point in both Reflect and Morph scale sweeps.

The complete machine-readable reports are available under results/architecture_audits/.

Zero-shot evaluation

The four multiple-choice tasks were evaluated zero-shot with lm-evaluation-harness==0.4.12, seed 3407, and no few-shot examples. WikiText-103 was evaluated on the raw test split with a 1,024-token context and a 512-token stride.

Language-model evaluation

Raw reports, generation samples, exact standard errors, environment metadata, and timing information are stored in results/.

Quick start

This release uses an exact custom PyTorch implementation. It is not presented as a native Transformers AutoModelForCausalLM checkpoint.

# Run these commands inside the downloaded model repository.
pip install -r requirements.txt
python verify_release.py

Generate a continuation:

python inference.py \
  --prompt "The future of artificial intelligence" \
  --max-new-tokens 128 \
  --temperature 0.8 \
  --top-p 0.95

Python usage:

from inference import generate, load_model

model, tokenizer = load_model(".")

text = generate(
    model=model,
    tokenizer=tokenizer,
    prompt="Science is important because",
    max_new_tokens=128,
    temperature=0.8,
    top_p=0.95,
    seed=3407,
)

print(text)

The implementation automatically uses CUDA BF16 when CUDA is available and FP32 on CPU. Autoregressive generation does not yet implement a KV cache, so GPU generation is recommended.

Release contents

Training data

The tokenizer and model were trained from scratch on an English mixture assembled from openly available sources. The mixture includes educational web text, mathematics and science material, academic and biomedical text, reference works, books, educational question-answer content, and WorldPairs.

Examples of upstream collections used during corpus construction include FineWeb-Edu, FineMath, peS2o, PubMed, arXiv abstracts, Cosmopedia, LibreTexts, Wikimedia, Project Gutenberg, and StackExchange. The full training corpus is not redistributed in this repository, and every upstream component remains subject to its original license and terms.

Intended use

Suitable uses include:

• research on language models under strict parameter and compute budgets;
• studying recurrent depth, low-rank correction, and sparse token routing;
• reproducible base-model evaluation;
• continued research training or downstream experimentation.

Limitations and safety

• This is a base model, not a conversational assistant.
• It may hallucinate facts, repeat text, or produce incoherent continuations.
• It can reproduce biases or unsafe patterns present in its training data.
• It has not been trained for safety refusal, instruction following, tool use, or retrieval.
• Its 1,024-token context and 48.94M-parameter capacity limit long-range reasoning and factual coverage.
• It must not be used as the sole basis for medical, legal, financial, or safety-critical decisions.

Reproducibility and integrity

The checkpoint is stored in SafeTensors format. For compatibility, both token_embeddings.weight and lm_head.weight are serialized; the runtime model ties them to the same parameter, so the unique trainable count remains 48,944,657.

Run:

python verify_release.py
sha256sum -c CRITICAL_CHECKSUMS.sha256

Expected critical hashes:

model.safetensors  91db1ccda91615523e21e57eb0a0e2d5cfba8a4f0ee5dae3eaef96f387a968d5
tokenizer.json     9502f89628bbe79d6adf98c0e199bf3ac4c74864d8175674da45ef039e38c44e

License

The model code and released weights are provided under the Apache License 2.0. Dataset components retain their upstream terms.