QuadOrbit-40M
QuadOrbit-40M is the public research checkpoint evaluated in the paper "QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models." It adds a small bounded complex recurrent memory branch to each Transformer block while retaining causal attention and SwiGLU layers.
This is a base next-token language model. It is not instruction tuned and it should not be presented as a production chatbot.
Model details
| Property | Value |
|---|---|
| Parameters | 39,999,240 |
| Hidden width | 512 |
| Layers | 8 |
| Query heads | 8 |
| Key/value heads | 1 |
| Orbit width | 8 per layer |
| Context length | 512 tokens |
| Vocabulary | 32,768 byte-level BPE tokens |
| Training step | 2,999 |
| Seed | 2026 |
The uploaded weights are the renamed
orbitoid_v2_stable_complex_orbit_attention_lm checkpoint. The mathematical
model is unchanged. Only the public name is now QuadOrbit.
Usage
Install the dependencies:
pip install "torch>=2.4" "transformers>=5.0" "tokenizers>=0.20" "safetensors>=0.4"
Load and generate:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Argo1-OOAS/QuadOrbit-40M"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device)
inputs = tokenizer("The future of language models", return_tensors="pt").to(device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=50,
do_sample=True,
temperature=0.8,
top_k=50,
use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
The repository contains custom model code, so loading requires
trust_remote_code=True. Review modeling_quadorbit.py before loading remote
code. This portable release uses the clear PyTorch recurrence and does not
require Triton. Generation is faster on a CUDA GPU.
Training data
The model was trained from scratch on a 209,715,200-token mixture:
| Source | Share |
|---|---|
| FineWeb-Edu | 45% |
| DCLM-Edu | 25% |
| Stack-Edu | 15% |
| FineMath-4+ | 6% |
| InfiWebMath-4+ | 5% |
| Cosmopedia v2 | 4% |
Training used 3,000 updates with 262,144 sampled tokens per update, or about 786.4 million token presentations. Because the prepared corpus contains 209.7 million tokens, examples were sampled more than once during training.
Evaluation
The full validation evaluation used 104,448 consecutive target tokens in FP32.
| Model | Validation loss | Perplexity |
|---|---|---|
| Parameter-matched Transformer | 3.7383 | 42.02 |
| QuadOrbit-40M | 3.7398 | 42.09 |
The difference is small and comes from one training seed. It does not establish
an improvement over the Transformer at this scale. See quadorbit.pdf for the
full method, smaller-model result, proof, and limitations.
Intended use
This release is intended for architecture research, reproducibility, analysis, and small-scale experimentation. Users may study the recurrence, reproduce the reported evaluation, or continue training under the license terms.
Limitations
- It is a small base model and is not suitable for reliable factual assistance.
- It has not been safety tuned or instruction tuned.
- It may generate incorrect, biased, repetitive, or offensive text.
- The context length is limited to 512 tokens.
- The evaluation uses one corpus and one seed.
- The portable recurrence does not implement a generation KV cache.
Do not use this model for medical, legal, financial, safety-critical, or other high-impact decisions.
Licenses
The model weights and repository code are licensed under the Apache License 2.0. The included research paper is licensed under CC BY 4.0. Dataset content is not redistributed here and remains under the terms of its original providers.
Citation
@misc{argo1ooas2026quadorbit,
title = {QuadOrbit: Bounded Quadratic Complex Recurrence for Hybrid Language Models},
author = {Argo1-OOAS},
year = {2026},
url = {https://huggingface.co/Argo1-OOAS/QuadOrbit}
}